← back
How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed
Takeaway
Train specialized coding models with repaired teacher data, carefully filtered user outcomes, and production validation.
Summary
- Zeta2 distills a small edit-prediction model from opt-in editor snapshots containing nearby code, recent edits, definitions, and diagnostics.
- Teacher predictions pass heuristic checks and a repair stage before experiment-specific formatting; the reusable pipeline stores stages as JSONL.
- Settled user edits provide noisy feedback, filtered by sampling inexpensive student predictions and comparing their distance to the eventual code.
- Held-out tests track reversal and edit-quality metrics, while production experiments measure acceptance and latency because offline scores may not match user preferences.
edit-predictiondistillationzed
Original description
To validate settled data, Zed ran 10 frontier model predictions per example and measured Levenshtein distance to the final state. For 100,000 training examples that is a million frontier model requests, which is prohibitively expensive. The fix: Zeta 2's student model now approaches teacher quality, so they run it 50 times instead at negligible cost. Ben Conungle, edit predictions lead at Zed, walks through how this pipeline came together. The pipeline pulls opt in production edit traces, distills them through a frontier teacher, and routes bad predictions through a repair step before formatting for the student. The ideal training examples sit in the middle of the Levenshtein distance distribution: too close to the settled state is obvious, too far is noise. A metric called reversal ratio, how often the model undoes exactly what the user just typed, was the key diagnostic for catching bad model behavior before shipping.