What an eval cannot tell you
An offline eval scores the model’s output against a dataset, which means it is silent about every failure that lives in the handoff — and the handoff is the entire surface you were hired to design.
The extraction eval is passing at 0.94 and the room relaxes. Somebody puts it on a slide next to the review-gate work, and the implication sits there unstated: the model is good, so the surface around it is fine.
You need to be able to say precisely what that 0.94 covers. Not to attack it, and not because it is wrong. Because an eval score and a correction rate answer different questions, and the fastest way to lose an argument about your own surface is to let a number from somebody else’s system stand in for one from yours.
What an offline eval actually is
Three parts, and every vendor names them roughly the same way. Braintrust’s own guide is the clearest decomposition: an eval is data, a task and scorers — test cases with inputs and optional expected outputs, the function under evaluation, and the functions that grade what came back. Braintrust sells an eval platform, so that is a vendor describing the shape of its own product; the decomposition is uncontroversial enough that this does not matter much, but the habit of saying it out loud does.
LangChain’s LangSmith documentation adds the deployment axis. Offline evaluation uses datasets for benchmarking and regression testing; it runs before you ship, against examples you curated, with a reference to compare against. That is what a number like 0.94 usually is: this task, over these examples, scored by this grader, on this commit.
Every one of those four qualifiers is load-bearing, and the score carries none of them. A field-level extraction eval over two hundred clean scans is a different measurement from the same eval over two hundred phone photographs, and both are reported as one decimal.
Online evaluation, and the admission inside it
The same LangSmith page describes online evaluation as running on production traffic to monitor real-world performance. Braintrust describes its own version and, in doing so, names the constraint that makes online eval hard: live requests have no ground truth. There is no expected output for the document a customer uploaded four minutes ago, so the reference an offline eval compares against does not exist. What gets substituted is a model grading the output instead — which is the entire subject of the judge lesson, and the reason it comes next.
Hold on to the admission rather than the mechanism. Both vendors are telling you, in their own marketing surface, that the moment evaluation moves to live traffic it stops comparing against truth and starts comparing against an estimate.
The gap, stated as this course’s own argument
Neither of those pages makes the claim this course cares about, so it is labelled as the course’s own reasoning rather than dressed in a citation. An eval scores what the model produced. Every failure at the HouseWarm review gate happens after the model has already produced it.
Walk the gate and it is obvious. The agent returns eleven fields with per-field confidence, a source-image crop and the raw OCR text. An offline eval can tell you how often those eleven field values match a reference. It has no representation at all for:
- A reviewer who approved in three seconds without opening the crop.
- A confidence score displayed in a way that reads as reassurance rather than as a hedge.
- A field that was edited, then reverted, then approved.
- A run abandoned halfway with the tab still open, which HouseWarm cannot resume.
- An escalation path nobody uses because using it looks like admitting you could not decide.
None of those are model failures. All of them are failures of the thing you were hired to design, and every one of them is compatible with a 0.94.
Now the other direction, because overclaiming runs both ways
The tempting next move is to say the eval does not matter and only the interface does. That is the same error with the sign flipped, and it is the one that will actually get you dismissed in the room.
An offline eval does things your gate structurally cannot. It runs against the same examples on every commit, so a regression is attributable to a change rather than to whoever happened to be on shift. It has a reference, so it can be wrong in a checkable way. It costs nothing per run relative to a human, so it can cover cases your reviewers will see four times a year. And it answers the one question the gate is genuinely blind to: whether the model got worse, as distinct from whether the people got tireder.
The relationship is complementary, and stating it that way is what earns you the right to ask the eval owner for something.
Two numbers that are allowed to disagree
Put an eval score and a correction rate side by side and the four combinations are all informative. This table is the course’s own framing, not a published matrix:
| Eval score | Correction rate | What is worth asking |
|---|---|---|
| High | High | Production inputs do not look like the eval dataset, or reviewers are correcting things that are not errors. |
| High | Low | The flattering reading, and the one to distrust: it is also what disengagement produces. |
| Low | High | The system is working as designed. The gate is catching what the model got wrong, which is what a gate is for. |
| Low | Low | The most alarming cell on the table. Bad output being approved anyway. |
Only the bottom-left cell is comfortable, and it is the one that gets read as failure. That inversion is worth carrying into a review: a gate doing its job produces numbers that look like the model doing badly.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
The eval owner says the extraction model is at 0.94 and asks why you need any gate metrics at all. What is the shortest honest answer?
Check your answer
That the 0.94 is a measurement of the model’s output against a reference dataset, and every failure mode you are responsible for happens after that output exists. A reviewer who approves without opening the source crop, a field edited and reverted, a run abandoned mid-review: the eval has no column for any of them, because they are not properties of the output.
Then give something back rather than only taking. The gate produces labelled corrections the eval dataset does not have, and those corrections are the one thing that can tell the eval owner whether their dataset still resembles production. That trade is the subject of the corrections lesson, and offering it is what turns this from a boundary dispute into a pipeline.
Hands on
Fill section 4 with the questions that are not yours
Done when: Section 4 of INSTRUMENTATION.md has at least five rows, each naming a question, the role that owns the answer, and the metric it blocks — and every row is a question about evaluation that you have deliberately not answered yourself.
- Open
learning/agent-evaluation/INSTRUMENTATION.mdat section 4, open questions. - Write the questions the eval owner has to answer before your metrics mean anything. Start with these: what is in the eval dataset, when was it last refreshed against production inputs, and is the per-field confidence the gate displays calibrated or a raw model score.
- For each, name the role rather than a person — eval owner, data owner, privacy reviewer — and name the metric that stays uninterpretable until it is answered.
- Add one row going the other way: what your gate can give the eval owner that they cannot get from their own dataset. Do not specify how yet. The corrections lesson is where that becomes a design.
- Reread the section and delete any row you could answer yourself. A question parked in section 4 that you were capable of resolving is just deferred work with a professional-looking table around it.
What this does not cover
This lesson stopped at the point where online evaluation substitutes a model’s judgement for a missing reference, and asserted nothing about how reliable that substitute is.
The judge lesson takes it from there: what a model grading a model gets measurably wrong, which of those failure modes have been directly measured and by whom, and the one that the most-cited paper on the subject explicitly could not confirm. The tooling lesson then puts LangSmith, promptfoo and Braintrust next to each other so that your preference among them comes from running one rather than from reading three landing pages.
Read this next — primary source
LangSmith — evaluation conceptsLangChain — vendor documentation, fetched 2026-09-05. LangChain writes this page and sells the hosted platform it explains, so treat every capability claim on it as the vendor describing its own product.
This lesson takes one structural distinction from it: offline evaluation runs a dataset of examples through a task and scores the output against a reference, while online evaluation runs on production traffic to watch real-world performance. Read it whole for the vocabulary, because it is the vocabulary the person who owns the eval stack will use in front of you: dataset, example, task, scorer, annotation queue, trace. Where it stops is the point of this lesson. Nothing on the page describes a human review interface, because the product it sells does not measure one.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.