Every number has two readings
A falling correction rate means the agent got better or the reviewer stopped reading, and the data cannot tell you which — automation bias is indistinguishable from success unless you instrument for the difference on purpose.
Correction rate at the review gate has fallen from 40% to 12% over two quarters. You are in front of the portfolio company that funded the work. The slide says the extraction agent got dramatically more accurate.
The slide may be right. It is also exactly what the data would look like if the agent had not improved at all and the reviewers had simply stopped reading — if approve had become a rhythm rather than a decision, three seconds per document, eleven fields unexamined. Both stories produce a falling correction rate. Both produce faster reviews. Both produce a rising acceptance rate and a time-to-trust curve that bends in the direction everyone hoped it would.
Automation bias is not a signal you can distinguish from success. In the data as specified in the definitions lesson, it is indistinguishable from success. That sentence is the centre of this course. Everything the instrumentation module does afterwards exists to make it less true.
The second reading, for each of the five
Take the metric set and write the uncomfortable interpretation next to each. Not the edge case — the ordinary, likely, competing explanation:
- Correction rate falls. The agent improved, or the reviewer disengaged. Same number.
- Acceptance rate rises. Output got better, or the reviewer’s threshold for “good enough” dropped, or the interface made approving cheaper than scrutinising. Same number.
- Time-to-trust shortens. The system earned trust faster, or it invited unearned trust faster. The construct cannot tell those apart — it never observed trust, it observed a correction rate falling, which is the first line of this list again.
- Mid-review abandonment falls. The gate got easier to finish, or the reviewer stopped hesitating over cases they should have hesitated over. Hesitation-then-completion and no-hesitation look identical in a completion metric.
- Escalation rate falls. Fewer genuinely ambiguous cases, or the escalation path acquired a social cost and reviewers learned not to use it. Escalation is the metric most sensitive to things that have nothing to do with the agent.
Notice the shape. In every case the flattering reading is a claim about the model and the alarming reading is a claim about the human, and the metric was computed from the human’s behavior. You are measuring the reviewer and reporting the agent.
The reviewer is not a reliable error-catcher, and this is measured
This is the part it is tempting to soften, so here are the numbers rather than the adjective. A systematic review in the Journal of the American Medical Informatics Association screened 13,821 papers down to 74 studies of automation bias in clinical decision support, and its meta-analysis reports a risk ratio of 1.26 (95% CI 1.11 to 1.44) for following erroneous advice: with a decision-support system in the loop, the risk of an incorrect decision rose by 26%. The same review gives the definition worth memorising — users over-accept computer output as a heuristic replacement for vigilant information seeking and processing — and splits the failure in two: errors of commission, following incorrect advice, and errors of omission, failing to act because the system did not prompt you to.
That split matters for instrumentation, because a review gate can only see commission. An omission error leaves no trace: the field the agent never flagged is the field the reviewer never looked at, and nothing in the interaction log distinguishes it from a field that was checked and found correct. Your correction rate is computed entirely from the error class your surface happens to be able to observe.
Ben Green’s survey of 41 human-oversight policy documents assembles the harder evidence, and its conclusion is blunt: the vast majority of research suggests that people cannot reliably perform the desired oversight functions. Two findings he collects are worth carrying:
- Reporting on a live facial recognition deployment, police in London “overwhelmingly overestimated the credibility” of the system, judging computer-generated matches to be correct at three times the actual rate of accuracy — while under no obligation to follow its advice.
- Automation bias persists after training and after explicit instructions to verify the automated system. Telling reviewers to be careful is a documented non-intervention, which means “we trained the reviewers” is not an answer to any of this.
The regulation has caught up with the literature, which is unusual and useful. Article 14(4) of the EU AI Act requires that a person assigned human oversight of a high-risk system be enabled to remain aware of the possible tendency of automatically relying or over-relying on the output — and it names the term, “automation bias,” in the statute itself. A design obligation, in a law, aimed exactly at the ambiguity in your metric. (That link is the AI Act implementation project’s reading copy of Regulation (EU) 2024/1689 rather than the Official Journal.)
The trap you are most likely to build yourself
The instinct of a good design-systems engineer, handed this problem, is to explain more. Show the confidence. Show the source crop. Show the raw OCR. Show why the model said what it said. HouseWarm’s gate already does all four, and it is a genuinely good interface. The uncomfortable finding is that explanation is not the lever it feels like.
Bansal and colleagues ran 1,626 crowd workers across three tasks — beer-review sentiment, Amazon book-review sentiment, and twenty LSAT logical-reasoning questions — with the AI’s accuracy deliberately pinned at 84%, roughly human parity, so that deferring to it could not masquerade as improving. Their headline, verbatim:
“explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness” (Bansal et al., CHI 2021)
The comparison underneath it is the one to carry into a design review. On the beer task, a team given only a bare confidence score reached 0.89 ± 0.05 accuracy; a team given confidence plus a generated explanation reached 0.88 ± 0.06 — no significant difference. Neither richer explanations, nor adaptive ones, nor expert-written ones instead of model-generated ones moved it. The explanation UI, in that study, did no work the confidence number was not already doing.
And the mechanism behind the flat line is the thing that should genuinely worry you: the authors report that explanations often increased accuracy when the AI was correct but decreased it when the AI erred, producing a minimal net change. Two opposite effects cancelling. An aggregate accuracy dashboard shows a flat line while the interface is actively making the failure cases worse.
That is not an argument against showing evidence in a review gate. It is an argument that “we show the evidence” is not itself a finding, any more than “the users like it” is, and that an aggregate metric can be perfectly stable while the distribution underneath it moves in two directions at once.
What actually separates the two readings
If the readings are indistinguishable in the metrics you have, the move is to find the signal where they differ. The most useful pointer comes from Buçinca, Malaya and Gajos, who ran 199 analysed participants on a nutrition task against a simulated AI held at 75% accuracy, comparing ordinary explainable-AI interfaces against three cognitive forcing functions — hide the suggestion until the user asks for it; require the user to commit to an unaided answer first, then reveal the AI’s; or make the user wait while the AI “processes,” so they form their own hypothesis. On the sub-decision they analysed most closely, overreliance on incorrect predictions fell from 0.64 to 0.48.
The finding that belongs in your instrumentation spec is not that one, though. It is this: self-reported trust was significantly positively correlated with overreliance on incorrect model predictions, while mental demand was positively correlated with performance on exactly those cases. Higher measured trust predicted worse outcomes where it mattered. Friction predicted better ones.
Read that as an instruction about which events to emit. The disambiguating signals are the ones a completion-shaped metric throws away: how long the reviewer dwelt before approving, whether they opened the source crop at all, whether a field was edited and reverted, whether they scrolled to the evidence before the approve. Those separate the agent got better from the reviewer stopped looking, and none of them reaches a backend that only hears about approvals.
Two honesty obligations from that paper
First, the ceiling. The same study reports that participants with no AI assistance at all outperformed every AI condition — forcing functions included — on the cases where the model was wrong. Cognitive forcing narrowed the damage; it did not remove it.
Second, the equity result. The forcing functions helped participants high in need for cognition significantly more than those low in it, and the authors frame this explicitly as an audit for intervention-generated inequality. An intervention that works on the reviewers who were already engaged is not a fix for the reviewers you were worried about.
Where the flattering reading actually costs you
Microsoft’s Aether group — the company’s own AI ethics committee, so a vendor-affiliated lab reviewing a problem in products Microsoft ships — collects one asymmetry in its overreliance literature review that explains why this is a professional risk and not just an intellectual one: user trust in AI drops by a relatively large amount when system capability decreases, and comes back by a much smaller amount when capability recovers. Trust is expensive to lose and slow to rebuild.
The same asymmetry applies to you. Present a falling correction rate as proof the agent improved, and be wrong once in a way somebody eventually discovers, and every subsequent number you bring gets discounted — including the true ones. Naming the second reading in the same breath as the first is not intellectual honesty as a virtue. It is the only version of this job that survives contact with a second quarter.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Why is “we trained the reviewers to be vigilant” not an acceptable answer to the ambiguity in these metrics?
Check your answer
Because it has been tested and it does not hold. Green’s review reports that automation bias persists even after training and after explicit instructions to verify the automated system. Training changes what reviewers say they will do; it does not reliably change the deference measured in their behaviour.
It is also the wrong shape of answer. Training is an intervention aimed at the human, offered in place of a measurement. Even if it worked, it would not tell you which of the two readings your falling correction rate has — it would just give you a reason to prefer the flattering one. The thing that resolves the ambiguity is a signal the two readings disagree about, emitted by the interface, and that is a design decision, not a training programme.
Hands on
Write the second reading for all five
Done when: Every row in table 2 of INSTRUMENTATION.md has a filled Second reading and a filled “What would distinguish the readings” — and at least three of those distinguishing signals are things the client can observe and the backend cannot.
- Reopen
learning/agent-evaluation/INSTRUMENTATION.md, at the two columns the definitions lesson told you to leave blank. - For each of the five metrics, write the second reading as a claim about the reviewer, in the same register as the first — not hedged, not labelled a risk. If the first reading is “the agent got more accurate,” the second is “the reviewer stopped reading,” not “there is a possibility of reduced engagement.”
- Then, per row, name the one signal you would need to tell them apart. Not three signals — one, the cheapest one that the two readings genuinely disagree about. Write what it is and, roughly, what component would have to emit it.
- Mark which of those signals exist today in a gate you have actually built. For each that does not, add a line to section 4 — open questions — rather than quietly assuming it can be added.
- Write one sentence, at the bottom of the section, that you would say out loud when presenting a favourable correction-rate trend. It has to be a sentence you would be willing to say in front of the person paying for the work.
- Bring it into the chat. I will take the flattering reading in every row and argue for it, and your job is to name the signal that would settle it.
What this does not cover
You now have five defined metrics and, for each, the signal that would disambiguate it. What you do not have is any of those signals, because nothing in the interface currently emits them — and a signal named in a document is not a signal.
The schema lesson picks that up: writing the event schema before the component, so dwell, revert, evidence-opened and abandonment become constraints on how the gate gets built rather than a retrofit onto whatever shipped. It is followed by the lesson on what the client can observe that the backend cannot, which is where most of the disambiguating signals you just named will actually have to come from.
Read this next — primary source
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team PerformanceGagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro & Daniel S. Weld, CHI 2021 — free author copy from the UW Interactive Data Lab; the ACM Digital Library version is paywalled
The lesson takes the headline result and one comparison. The paper is worth reading whole for its method, which is the actual contribution: earlier work found explanations helped, but always with an AI substantially better than the human, so any measured gain could just be the human learning to defer. These authors pin the AI at 84% — roughly human parity — precisely so that deference stops looking like improvement. That design decision is the transferable idea, and it is the same move you have to make when instrumenting a review gate: build the measurement so that the flattering explanation and the alarming one produce different data.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.