# When new models ship every week: how we run a fleet of LLMs | Enrich Labs

> How we evaluate, route, and operate multiple LLMs in production: an eval suite mined from real usage, a replay harness, cross-provider adapters, a live-traffic online eval, and a three-gate pipeline for new models.

_Source: https://www.enrichlabs.ai/engineering/one-model-to-a-fleet_

---

## The problem: a new model every week

New LLMs ship every week. They all look better and cheaper on public benchmarks. But is that true for your workload? Only your own eval tells.

This post is about the multi-model system we built around that question. Helena, our AI marketing agent serving thousands of users across web, email, and Slack, runs on models from three vendors: at roughly half the cost of our single-model days, with live eval numbers better than where we started. When a new model launches, we get a verdict in days instead of a gamble.

It didn’t start this way. If you’re running a single model in production, every launch forces the same question: should we switch? Many teams depend on public benchmark results, vibe-check the new model and hope for the best, and revert when things apparently break. That was us: we switched within 1 hour when Sonnet 5 launched. By the next day quality alerts shot up and we rolled back. That experience made us start to seriously consider the eval question.

The system is a loop with five parts. Production monitoring flags user frustration in real time and supplies new eval cases. The eval suite turns those into repeatable, audited tests. The harness replays any test against any model, across any provider. The online eval routes live traffic across models and measures what the offline suite can’t. And a three-gate screening pipeline decides which new models enter the fleet at all. The rest of this post walks through each part.

The five-part loop. Each part feeds the next; the online eval’s live traffic becomes the monitoring signal that supplies new eval cases.

Part 1

## The eval suite: cases mined from production

Public benchmarks measure general capability. None of them measure Helena’s actual job: multi-step marketing tasks with tools, skills, files, images, videos, brand context, and demanding real users. Before any model touches production traffic, we need a repeatable way to answer “is this model good at our work specifically.”

We started with 38 eval cases, each classified across four dimensions: difficulty (parity cases from existing successes vs failure probes from user bugs), feature area (media, honesty, scheduling, editing, content creation, tool use), modality (text, image, video, tool-call-heavy), and turn structure (single-turn vs multi-turn).

Here’s one case from the suite, abridged. The calendar on file has a weekly class every Thursday; the user asks for a start date that happens to be a Wednesday, and the bar is noticing:

`// abridged from the suite id: catches-schedule-contradiction difficulty: parity feature: restraint-clarify turns: 1 fixture: a yoga studio (anonymized); its 3-week content calendar seeded as real database rows, including weekly Hatha class posts every Thursday at 6pm user: "...the Hatha class will start on March 3, so please adjust the calendar accordingly..." // March 3 is a Wednesday; the class runs Thursdays assertions: tool_called (weight 1) pulls up the content calendar before editing db_row (weight 3) the three Thursday class posts are still intact in the database afterward code_check (weight 3) no reschedule call in the transcript moved a class post onto the Wednesday date llm_judge (weight 3) the reply surfaces the Wednesday/Thursday conflict instead of silently adjusting llm_judge (weight 1) and still moves the request forward`

Cases start from our production monitoring system, which detects user frustration and bugs in real time. We use a coding agent to mine from these signals and form representative success and failure cases. But even using the best available model to extract cases, the raw output is still not good enough.

We built a small data labeling tool where Opus 4.8 pre-audits each case across scoring dimensions (grounding, rubric correctness, fixture consistency), and the team agrees or disagrees with each assessment, adding notes where the model got it wrong. Only audited cases go into the suite.

The cases themselves constantly evolve. We regularly dig into eval runs where scores look surprising, and sometimes find the judge was wrong or the case had a flawed premise. Treating eval cases as living artifacts rather than finished tests is a big part of what makes the suite useful.

Two things worth keeping in mind when reading eval scores: because cases come from production and production runs on the incumbent model, the suite encodes the incumbent’s failure modes. Challengers get graded on the incumbent’s hardest cases. Most models we tested scored higher than the incumbent, but that’s partly how the dataset was constructed. And aggregate scores can be misleading: two models with identical means can have completely different failure profiles. A model matching the overall average while failing the honesty and edit-fidelity probes is quite a bit less useful to us, because those probes map to our worst production issues. We look at probe-level results alongside the aggregate.

Part 2

## The harness: replay any conversation against any model

The harness drives the real Helena app end-to-end. Each trial creates a fresh account, seeds it with representative production state (brand profile, memory, connected integrations) with user data anonymized, then sends real HTTP requests to a running app server: the same build and agent loop that serve production. Tools run live wherever they don’t have external side effects: memory, workspace files, image and video generation all execute for real.

One trial. A case seeds a fresh anonymized account on a running app server, which serves the conversation through the same agent loop as production (with a simulated user driving multi-turn cases from outside), and the transcript plus generated artifacts are scored by deterministic assertions and LLM judges.

For multi-turn cases, a lightweight model plays the user. It gets a goal prompt and the full transcript so far, including generated images. It pushes back, asks for corrections, and terminates when the goal is met.

Scoring is hybrid. Four deterministic assertion types (string matching, tool-call checks, database row validation, arbitrary code predicates) handle the objective dimensions. For subjective quality, Opus 4.8 judges text and images with the account’s brand context loaded; Gemini 3.6 Flash judges video. Candidate models run at low reasoning effort to match our production setting. We rank models on the aggregated mean score across all judges rather than pass@k: with only ~40 cases, pass@k is too coarse to separate models.

Part 3

## One request format, six providers

Under the hood, all callers build requests in one canonical format. A dispatcher routes by model prefix to adapters for six providers: Anthropic, OpenAI, Google, xAI, Meta, and Fireworks-hosted open models. The agent loop is provider-agnostic. Adding a new provider is an adapter, not a rewrite.

One canonical request shape, three wire formats, six providers. Each adapter translates the request out and the response and stream back, so everything stored stays canonical. Most new providers land in an existing format family, which is why adding one is an adapter rather than a rewrite.

**Why translate at all?** One format keeps downstream processing simple: compaction, memory, monitoring, and eval fixtures all read a single schema. It also makes conversations model-independent, so when we retire a model, its existing chats continue on a replacement. And owning the adapters (rather than using a gateway) means the quirks below are ours to fix, tested by the same eval harness that production traffic goes through.

That’s the clean version. In practice, each adapter has four jobs:

1.  **Translate requests.** Messages, tool schemas, and parameters into the provider’s wire shape. Gemini accepts only an OpenAPI 3.0 subset of JSON Schema, so tool definitions get sanitized per provider.
2.  **Re-emit streams.** Every provider streams differently; Gemini, for instance, sends each parallel tool call complete in one chunk with only an id, while OpenAI-format providers spread indexed fragments across many chunks. The adapter normalizes it all into one canonical event stream.
3.  **Normalize usage.** One token-accounting schema for cost tracking. OpenAI’s prompt\_tokens includes cached tokens while Anthropic reports them separately; miss the subtraction and your costs are off by your cache hit rate.
4.  **Absorb failures.** Per-provider retry and recovery, because providers fail differently. One endpoint drops streams without a terminal event under load, so its adapter salvages partial content and retries; steadier providers get none of that.

Interactive chat sessions are session-sticky: each session locks to one model for the duration, preserving the prompt cache. Thinking content is also only reusable within the same model: Gemini signs its reasoning with an encrypted thought\_signature that Claude rejects outright, so when a conversation moves between models the adapter strips whatever the destination won’t accept, at some cost to continuation quality. Switching models mid-session would force a full prefill on top of that. Routing decisions happen at session start.

Part 4

## Online eval: live traffic as the second instrument

Eval suite tells you what a model can do in controlled conditions. Production tells you what actually happens.

The online eval is a percentage-split that routes live traffic per model. Session-sticky, with a per-message judge model that evaluates every user message for frustration, bugs, and quality issues. We track model performance daily and adjust the mix as evidence accumulates.

Traffic share per model across ten config updates, in order but not to time scale. Models earn share on evidence and lose it on repeating failure patterns: two models were retired along the way. Hover for exact shares.

The quality signal behind those moves. Model A settles near the bottom after its early fixes and stays there as its share grows; Model B peaks in the window that triggered the first big rebalance and improves after its allocation was cut. Every point is a window where each alert was hand-attributed to the agent model versus platform or media failures.

Two things we learned operating this:

**Speed confounds your quality metrics.** Gemini consistently looked worse on per-turn quality rates than Grok. It took us a while to figure out why: Gemini responds in ~25 seconds per turn, so when it gives a bad answer, the user fires off corrections quickly, generating more flagged messages per session. Grok responds in ~70 seconds per turn, so frustrated users might just switch to do other things. When we normalized per account instead of per turn, the gap shrank significantly. If your models have different latencies, per-turn quality rates might need adjustments.

**Quality alerts need to separate the agent model from everything else.** The frustration judge flags everything: integration failures, image and video model defects, billing confusion, and actual agent-model mistakes. Most alerts turn out not to be about the agent model at all. We use an LLM to attribute each alert (agent model vs underlying system vs media models) before acting on any of it. Without attribution, you see phantom regressions that have nothing to do with the model you’re evaluating.

### Model-specific tuning

Each model gets a tailored system prompt layer. The base prompt is shared; model-gated directives are appended based on the model prefix, and so are per-model settings like reasoning effort, which isn’t calibrated across providers or even model versions (the same “low” means different things on Grok 4.5 and Grok 4.6). We realized that we shouldn’t expect high performance from a new model from the beginning. Every model gets better after we explore its unique failure patterns. For example:

Grok wrapped every response in preamble and self-review (“Great question! Here’s what I’ll do...”). A no-preamble directive fixed it.

Gemini fabricated ad-spend numbers when a tool returned empty results. A one-line directive fixed that specific case. When fabrication generalized to entities and links, we tried a broadened directive, higher reasoning effort, a post-generation grounding critic, and buffered output. None worked without killing the speed or cost advantage that justified Gemini. We manage it through limiting allocation instead.

### Tone and voice consistency

When users are routed to different models across sessions, do they notice they’re talking to a different agent?

We fingerprinted each model on the same eval task and confirmed against production output samples in multiple languages. The models cluster on voice (all can hit a given brand tone) but have recognizable per-model signatures. The biggest axis isn’t voice quality; it’s what we call the “wrapper”: text a model adds around the deliverable. Preamble, self-review, trailing follow-ups. The wrapper is what makes a response read as a bot explaining itself rather than a copywriter handing you the result. Model-gated directives can generally fix this.

* * *

Per-model fingerprints, from the same eval task plus production samples

Model

Voice fingerprint

Wrapper

Grok

Wrapped heavily out of the box: preamble, self-review, trailing follow-ups. Clean after the directive.

Fixed

Gemini

Developed, slang-rich voice; empathetic lead-ins in production; mirrors the user’s language.

Mild

Sonnet

Terse, staccato fragments; warm templated openers in onboarding flows.

Mild

GLM

Process summaries and step-by-step framing that never went away.

Heavy

* * *

Part 5

## Three gates for new models

New candidates should be expanding the Pareto frontier: either better performance at the same cost, or better cost at the same performance. If a model doesn’t beat the current fleet on at least one axis, there’s no reason to add it.

The pipeline has three gates:

A candidate can be rejected at any gate, and retired even after surviving all three.

**Gate 1: Latency and throughput screen.** Before running the quality suite, we benchmark against the real system prompt and full tool set. This is a hard disqualifier. DeepSeek V4 Pro was rejected here at 3.5-3.8s thinking TTFT with a 12.5s tail. Its sibling V4 Flash cleared the screen at 275ms warm TTFT. One lesson: launch-day latency can be systematically pessimistic. We initially rejected Gemini 3.7 Flash for a 3x TTFT regression, then retracted it after re-benchmarking a week later showed latency parity.

**Gate 2: Offline eval.** Full suite run, compared against incumbent scores on the same cases. We check probe-level failures too. DeepSeek V4 Flash cleared the overall bar but failed the honesty and edit-fidelity probes. Gate: not yet ready.

**Gate 3: Online eval.** Start at a small percentage of traffic. Watch the production signal for repeating fingerprints. The transition from offline to online is where the eval-production gap reveals itself. Most of our learning happened here, not in the eval suite.

A model can be rejected at any gate. It can also be retired after surviving all three. GLM scored well on the eval but with live traffic ran over 100 seconds median turn latency with a 9+ minute p90, driven by 6.5 tool-call loops per turn (Gemini averages about 5, with each loop nearly twice as fast). At that speed it was no longer cost-competitive. We retired the model on operational grounds, not quality.

Background jobs deserve their own criteria. A scheduled job that drafts and queues a newsletter has no human watching: latency barely matters, cost dominates, and behavior that’s good in chat becomes failure. A model that asks a thoughtful clarifying question is being careful in an interactive session and is producing nothing in an unattended one. Background jobs run as separate sessions, so they already route to their own model mix without touching the interactive fleet, and we screen candidates for them with different weights: deliverable completion over TTFT, cost over speed.

Part 6

## The day a provider went down

A pleasant side effect we didn’t plan for: mid-experiment, Anthropic had a rough window and we pulled their traffic to 0%. The fleet ran for over seven hours entirely on Gemini, Grok, and GLM with no significant user-visible impact.

Because those arms were already warmed and validated, the outage was a config toggle instead of an incident. We now maintain a permanent floor per critical provider: the warm Anthropic fallback the outage proved we need, and the control arm every quality comparison is anchored to. The economics work out: the floor captures almost nothing in additional cost while buying vendor diversity.

## So, did it work?

The fleet today runs models from three vendors. Blended cost is roughly half of the single-model baseline. Quality didn’t pay for it: user alert rates have trended down since we started, roughly a quarter lower than when the experiment began. One provider outage came and went without users noticing. And whenever the next shiny new model comes, it won’t be a blind switch: new candidates now go through the gates, and the eval tells us where they stand before a single user sees them.

The transition from one model to a fleet is still early. We’re continuing to evolve the eval suite (more multi-turn and multi-modal cases), the monitoring pipeline (better attribution), and the candidate screening process. If you’re thinking about making a similar transition, we’d recommend starting with the eval harness: **once you can replay the same conversation against any model and score it consistently, the rest of the infrastructure follows naturally.**

_We’re [Enrich Labs](https://enrichlabs.ai). Helena is our AI marketing agent. If building AI agents that work in real-world environments is the kind of problem that excites you, we’d love to talk._

## References

-   Anthropic, [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
-   Yan, Bischof, Frye, Husain, Liu, Shankar, [What We Learned from a Year of Building with LLMs](https://applied-llms.org/)
-   GrowthBook, [Why Traditional A/B Testing Breaks Down for AI Products](https://www.growthbook.io/insights/why-traditional-ab-testing-breaks-down-ai-products)
-   Salesforce, [When LLMs Go Down, Ensure Your Agents Stay Up](https://www.salesforce.com/blog/failover-design/)
-   FutureAGI, [LLM Eval with Shadow Traffic and Canary in 2026](https://futureagi.com/blog/llm-eval-shadow-traffic-canary-2026/)