V156 pipeline health monitoring + complete cron inventory
Some checks failed
WEVAL NonReg / nonreg (push) Has been cancelled
Some checks failed
WEVAL NonReg / nonreg (push) Has been cancelled
Discovered 7 Ethica cron scripts running: 01:00 enrich-v4 600 10:00 16:00 enrich-searxng 500 11:00 23:00 richscraper 1000 03:00 Sun scraper-cnam 100 03:00 12:00 20:00 cron-scraper 04:00 10:00 16:00 22:00 enrich-ma 300 Daily theoretical capacity: 4800 records/day Full pipeline state: Data: 161733 HCPs 110657 emails (+6 since V153) Recent 7d: 25131 lines added/modified Pilot HIGH quality DZ MG: 3542 Maghreb ready: 103460 Consent tokens pending: 500 Ethica senders: 10 active Graph sends historical: 567384 Ready for Kaouther GO signal: emails available ✅ consent tokens ✅ campaigns templates ✅ send capacity 230k/day ✅ monitoring dashboard ✅ Missing only: Kaouther validation flow RGPD/DZ laws + activate campaigns L99 153/153 PASS (24 consecutive versions V125-V156) Chain V131-V156 complete Doctrines 0+4+13+14+95+100 applied (monitoring only)
This commit is contained in:
135
wiki/session-V156-pipeline-health-monitoring.md
Normal file
135
wiki/session-V156-pipeline-health-monitoring.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# V156 - Pipeline health monitoring + complete cron inventory - 2026-04-22
|
||||
|
||||
## Découverte V156: 7 cron scripts Ethica actifs
|
||||
|
||||
```
|
||||
01:00 /opt/ethica-enrich-v4.py 600
|
||||
10:00, 16:00 /opt/ethica-enrich-searxng.py 500
|
||||
11:00, 23:00 /opt/ethica-richscraper.py 1000 (2x)
|
||||
03:00 Sun /opt/weval-l99/ethica-scraper-cnam.py 100
|
||||
03:00,12:00,20:00 /opt/ethica-cron-scraper.py
|
||||
04:00,10:00,16:00,22:00 /opt/weval-l99/ethica-enrich-ma.py 300
|
||||
```
|
||||
|
||||
**Daily theoretical capacity: 4,800+ records/day**
|
||||
|
||||
## V156 État complet pipeline
|
||||
|
||||
### 🎯 Data State
|
||||
```
|
||||
Total HCPs: 161,733
|
||||
With email: 110,657 (+6 since V153)
|
||||
Added 24h: 7
|
||||
Added 7d: 25,131 (pipeline actively growing)
|
||||
```
|
||||
|
||||
### 🎯 Pilot Readiness
|
||||
```
|
||||
DZ MG total: 10,063
|
||||
DZ MG with email: 4,069
|
||||
Pilot HIGH quality: 3,542 ← Kaouther ready
|
||||
Maghreb ready: 103,460
|
||||
```
|
||||
|
||||
### 🎯 Send Infrastructure
|
||||
```
|
||||
Ethica senders active: 10
|
||||
Consent tokens pending: 500
|
||||
Consent opt-ins history: 17
|
||||
Campaigns: 2 (both draft)
|
||||
Graph sends historical: 567,384
|
||||
```
|
||||
|
||||
## Monitoring disponibles
|
||||
|
||||
### Dashboard endpoint live
|
||||
```
|
||||
GET https://weval-consulting.com/api/ethica-api.php?action=dashboard&token=ETHICA_API_2026_SECURE
|
||||
→ JSON with total_hcp, with_email, with_telephone, specialites, villes,
|
||||
recent_7d, pipeline status, last_scrape
|
||||
```
|
||||
|
||||
### SQL Views V155
|
||||
```
|
||||
ethica.medecins_pilot_verified_dz_mg (3,542 HIGH quality)
|
||||
ethica.medecins_pilot_ready_maghreb (103,460 safe)
|
||||
```
|
||||
|
||||
### Queries monitoring utile
|
||||
|
||||
```sql
|
||||
-- Pipeline activity last 24h
|
||||
SELECT COUNT(*) FROM ethica.medecins_validated
|
||||
WHERE created_at > NOW() - INTERVAL '24 hours';
|
||||
|
||||
-- Pipeline activity by day (last 7d)
|
||||
SELECT created_at::date, COUNT(*)
|
||||
FROM ethica.medecins_validated
|
||||
WHERE created_at > NOW() - INTERVAL '7 days'
|
||||
GROUP BY 1 ORDER BY 1;
|
||||
|
||||
-- DZ MG progress
|
||||
SELECT COUNT(*) FROM ethica.medecins_pilot_verified_dz_mg
|
||||
WHERE quality_score >= 60;
|
||||
|
||||
-- Enrichment logs (last run each)
|
||||
SELECT 'enrich-v4' AS script, MAX(created_at) AS last_run
|
||||
FROM ethica.medecins_validated;
|
||||
```
|
||||
|
||||
## Chaîne complète V131 → V156
|
||||
|
||||
23 versions, ZERO régression L99 (153/153 maintenu).
|
||||
|
||||
### Clés résultats
|
||||
|
||||
1. **Routing 100% + Playwright proof** (V131-V132)
|
||||
2. **4/4 hubs anti-orphan** (V133-V134)
|
||||
3. **Admin logging 4 sources + KPI cards** (V135-V146)
|
||||
4. **Session cleanup 2481+607 UPDATE** (V143+V148)
|
||||
5. **Ambre-deps cache x250 faster** (V144)
|
||||
6. **Ethica pipeline resurrected** (V149-V152)
|
||||
7. **Send infrastructure audit** (V153)
|
||||
8. **500 consent tokens + quality views** (V154-V155)
|
||||
9. **Pipeline health monitoring** (V156)
|
||||
|
||||
## Ready for Kaouther GO
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| Emails DZ MG | ✅ 3,542 HIGH quality |
|
||||
| Consent tokens | ✅ 500 pending |
|
||||
| Campaigns templated | ✅ 2 drafted |
|
||||
| Senders Ethica | ✅ 10 active |
|
||||
| Send capacity | ✅ 230k/jour |
|
||||
| Views SQL | ✅ Pilot ready |
|
||||
| Monitoring dashboard | ✅ Live API |
|
||||
| Cron enrichment | ✅ 4,800/jour theoretical |
|
||||
|
||||
**Il ne manque que le GO de Kaouther** (validation flow consentement RGPD/loi DZ).
|
||||
|
||||
## L99 153/153 PASS ✅ (24 versions consécutives V125-V156)
|
||||
|
||||
## Doctrines V156
|
||||
|
||||
- 0 Root cause (monitoring gap identifié)
|
||||
- 4 Zero régression (read-only, query only)
|
||||
- 13 Cause racine (cron inventory complet)
|
||||
- 14 Test-driven (live data verified)
|
||||
- 95 Traçabilité wiki complet
|
||||
- 100 Train release
|
||||
|
||||
## Chain V131 → V156
|
||||
|
||||
```
|
||||
V131-V134 Routing + Playwright + hubs
|
||||
V135-V146 Admin + logging + KPI
|
||||
V147-V148 Ethica audit + null→legacy
|
||||
V149-V152 Ethica pipeline resurrected + boosted
|
||||
V153 Send infrastructure audit
|
||||
V154 500 consent tokens + forensics (autre Claude)
|
||||
V155 Pilot-ready views + quality scoring
|
||||
V156 Pipeline health monitoring + cron inventory
|
||||
```
|
||||
|
||||
**Status Kaouther: READY TO LAUNCH ON GO.**
|
||||
Reference in New Issue
Block a user