The flywheel: turning live traffic into better models

Shipping a first LoRA adapter is straightforward. You write a few hundred examples, train for an hour, and it beats the base model at your task. Then it plateaus, and you hit the actual question: where does the next dataset come from, on a team where nobody has time to annotate conversations?

Harrir's answer is a loop that runs on its own traffic, with humans placed only where they are worth the most.

live traffic (LangGraph checkpoints)
   1  judge          every new turn since the last fine-tune
   2  admin review   optional human override on flagged turns
   3  amplify        one correction becomes eight synthetic conversations
   4  curate         dedup, drop ungrounded, cap the mixture
   5  fine-tune      LoRA SFT on an H100
   6  auto replay    advisory only, never blocks
   7  replay gate    champion vs challenger on held-out data
   8  promote        gated hot-swap into the live vLLM
   back to 1

Judging against corrected hindsight

A 32B judge grades every new turn. It sees the live system prompt, the live tool schemas, the conversation so far and the current turn, then returns one verdict: ok, or wrong with a proposed correction.

The important detail is what 'the conversation so far' means. Turns are judged against a progressively corrected prefix, so a later turn is evaluated as though earlier mistakes had already been fixed. Without that, one bad turn early in a thread poisons the evaluation of everything downstream, and you get a run full of correct turns marked wrong because they followed sensibly from an error.

Not every verdict becomes a row

What happens to a judged turn depends on both the verdict and whether the proposed correction survives execution.

• Judge call errored on a timeout or 5xx: no row is written at all, and the turn re-enters the backlog for the next run. An infrastructure hiccup must never become a training label.
• Verdict unparseable: recorded as failed and retired.
• Verdict ok: row goes active. Trivial no-tool turns like 'thanks' are kept for audit but excluded from training.
• Verdict wrong, correction grounds and validates: recorded as corrected and active. This is the row the whole pipeline exists to produce.
• Verdict wrong, correction fails its checks: recorded as failed and retired, because a broken fix is worse than no fix.

Proposed corrections are re-executed for real against the live tool services before acceptance. Mechanical checks veto on three conditions: answering without retrieval, a tool error, or a retrieval call returning zero hits. Softer problems like reply shape or a language mismatch are recorded as non-blocking style flags. And the judge never overwrites a row an admin has already touched.

One correction becomes eight

Reviewed corrections are the scarcest thing in the system, so each one is amplified. A generator writes eight new conversations whose final turn uses exactly the corrected tool decision, with plausibly varied arguments, spread across English, MSA, Iraqi dialect and Franco-Arabic.

Every variant is validated before it is written: the tool must match the parent's fix exactly, and non-side-effecting tool turns need both a JSON result and a grounded reply. Each variant inherits its parent's train/holdout split, which is the guardrail that matters most here. Without it, amplification quietly leaks holdout examples into training and every evaluation number afterwards is a lie.

The queue is the dataset table itself, gated by an explicit pending flag an admin sets. There is no separate queue collection, and legacy rows without the field are never implicitly picked up. Runs are rate-gated at 30 requests per minute on a sliding window, bounded at three concurrent calls, and idempotent: re-running replaces a parent's previous variants rather than accumulating duplicates.

Curation is where the bodies are buried

Assembling the mixture is the least glamorous stage and the one that decides whether the run is worth anything.

dedup            keep highest priority: gold > human-reviewed > judge-corrected > newest
drop             replies asserting an ungrounded 4+ digit number
cap              unreviewed judge-ok rows  <= 50% of the mixture
oversample       corrections x4   (x6 when quality_grade == gold)
cap              corrections      <= 25% of supervised tokens
cap              escalate-shaped  <= 10% of the pool

The ungrounded-number rule is the most specific. Any reply stating a four-digit-or-longer number that appears nowhere in the tool observations, arguments or customer message gets dropped. That rule exists because a hallucinated phone number made it past the judge and into a dataset, and pattern-matching on digit runs turned out to be a more reliable catch than asking a model to notice.

Two gates, and what each one proves

Every fine-tune ends with an automatic replay pass that scores tool match, argument match and output shape. It is advisory. The run lands regardless, marked as pending replay, because a cheap well-formedness check should never be able to block an artifact from existing.

The gate that decides anything is admin-triggered. It hot-loads the candidate into the live vLLM under a temporary staging name, scores the current live adapter against the candidate on a held-out split over the real production chat-completions path, then unloads the staging adapter unconditionally.

ship = (candidate.accuracy  - champion.accuracy  >= 0)
   AND (candidate.tool_acc  - champion.tool_acc  >= 0)

Promotion, and the thing that will bite you

Promotion goes through vLLM's runtime LoRA API over the same endpoint the app already uses. The candidate loads under a staging slot, a trivial completion smoke-tests it, and only then does the live name get unloaded and replaced. A dry run stops before the swap, which proves the adapter is loadable and servable with zero production impact. A live promote is rejected outright unless the caller explicitly asserts the replay gate passed.

Then the swap is verified by checking that the served name appears in the model list, rather than assumed from a 200 response.

8

synthetic variants per correction

25

corrections before a run triggers

0

restarts to promote an adapter

The loop's real output is not a better adapter. It is that improving the model became a routine somebody can run on a Tuesday, with a defined gate and a rollback, instead of a research project that needs a free week and a hero.