Ground truth → retrain → gate
Six simulated months of a monthly retraining pipeline on public data (Amazon MASSIVE sv-SE, 60 intents). Two pipelines see the same data: naive (ingest everything, always promote) and gated (versioned GT store with quality/budget refresh, batch screening, drift checks, curated regression suites, promotion gate, all tracked in MLflow).
Production model accuracy on each month's live snapshot
Month 3 is when a new product area (food ordering: takeaway + cooking) starts appearing in live traffic; month 5's curation batch comes from a vendor whose labelling has a bug (40% of records get a wrong intent from the same scenario).
Month by month (gated pipeline)
Drift checks: what each signal saw
Scenario-level PSI against the GT blend. The alert threshold is not a rule of thumb: it is the 95th percentile of PSI between the GT and same-size random samples of the GT itself (what "no drift" looks like at n = 400).
Curated regression suites (gated, candidate vs production)
120 high-agreement test utterances per product requirement. A suite fails the gate when the whole 95% paired-bootstrap CI of the change lies below −2 points.
What this shows
How it works
- GT store (SQLite): every record keeps source, month added, annotator-agreement quality and status (active / quarantined / retired). Each refresh writes a versioned snapshot (member list + content hash), so any model can be traced to the exact data it saw.
- Refresh policy: retire records below 2/3 annotator agreement; keep the GT within a 6,000-record budget by retiring the oldest records of the best-covered intents; never thin an intent below 60 records (the history small classes depend on).
- Batch screening: the production model scores each incoming curation batch; records it confidently disagrees with (p ≥ 0.6) are quarantined; if more than 15% are flagged, the whole batch goes back to the curators.
- Drift: PSI on true labels of a small labelled live snapshot, PSI on predicted labels (label-free), out-of-vocabulary rate and confidence, each against a null threshold.
- Gate: retrain a candidate; compare to production on the live snapshot and every curated suite with paired bootstrap CIs; promote only if nothing regresses. Every run is logged to MLflow with the GT version and hash.
Model: TF-IDF (word 1–2 + char 2–5 grams) + logistic regression, CPU only. Everything is precomputed by pipeline.py; the numbers on this page are read from results.json. One seed, one simulated timeline: treat differences inside the CIs as noise.