The citation that does not support the claim
A real URL attached to a sentence the page does not support is the failure mode citations are supposed to prevent and instead disguise — and the UI can make that specific failure findable rather than invisible.
This is the failure the previous two lessons were quietly assuming away. The anchor resolves. The preview opens. The passage is real, the URL is real, the source is a reputable one. The reviewer reads it and the page does not say what the sentence claims it says.
Nothing in the interface was broken. Every component did its job. The citation was a lie and the UI rendered it faithfully.
The measurement
Liu, Zhang and Liang evaluated four generative search engines — Bing Chat, NeevaAI, Perplexity.ai and YouChat — with human annotators, and reported that across them, “a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence.”
Read the second number again. Roughly a quarter of the citations those systems produced did not support the sentence they were attached to.
Now scope it properly, because this is a number that gets quoted far outside what it measured. It is four named products, as they existed when the paper was written — submitted April 2023, revised October 2023, published in Findings of EMNLP 2023. It is not a measurement of current systems, it is not a measurement of any retrieval pipeline you build, and it is not a rate you may print next to your own component. It establishes something different and more useful: that a citation attached to a sentence by a production system is a claim about support, that the claim is frequently false, and that the failure is common enough to be a design case rather than an edge case.
Two failures, not one
The paper’s vocabulary is the part worth carrying into your component, because the two numbers above measure different things and they need different UI.
- Citation recall asks whether every statement in the response is supported by some citation. It fails when a sentence carries no anchor, or carries one that covers only part of what it asserts. The failure lives in the gaps between anchors.
- Citation precision asks whether every citation actually supports the statement it is attached to. It fails on the anchor itself. The failure lives inside a citation that looks complete.
The two look nothing alike to a reviewer. A recall failure is an absence — hard to spot, because nothing is drawn where the citation should be. A precision failure is a presence — easy to spot if you check, and invisible if you do not, which is most of the time. This lesson is about precision. Recall is the reason the previous lesson insisted on a state for an uncited sentence.
What the interface can actually do about it
The component cannot verify a citation. That is worth saying plainly, because most of the tempting features here are ones that quietly claim it can. What the component can do is refuse to imply verification, make the check cheap, and capture the result when someone does check.
1. Cited and checked are different states, and only one is free
Give the anchor a state model with at least three values: cited and unchecked, checked and supported, checked and contradicted. Default to the first, because the first is what you have. This is the same rule that governs span status in the first module — a system that has made no claim must render as having made no claim, not as a success.
The trap is that the third state has to be reachable and visible. Marking a citation as contradicted is worth nothing if the mark is only visible to the person who made it, or disappears on the next run, or does not travel with the run into whatever record the team keeps.
2. The check has to cost seconds, not minutes
This is where the previous two lessons pay off. The preview puts the passage next to the sentence, and the span highlight, where you have a producer honest enough to supply one, narrows it to something comparable. Every second removed from that comparison converts some fraction of reviewers from skipping the check to performing it.
The one thing not to add here is a scroll. If checking requires the reviewer to move the sentence off screen to read the passage, they are comparing from memory, and comparing from memory is where a plausible non-match passes.
3. Disagreeing has to be a first-class action
When a reviewer finds a citation that does not support its sentence, they have produced the single most valuable piece of data your system will see that day: a labelled precision failure, found by a human, on a real run. Most interfaces throw it away, because the only affordance is a thumbs-down on the whole response.
Attach the control to the citation, not to the answer. What comes back is the sentence, the source, the passage, and the fact that a human judged them unrelated. That record is worth more than the correction.
4. An automated check reports the checker’s opinion
You can run an entailment or support classifier over each sentence-citation pair and render the result. That is a reasonable engineering choice. It is not verification, and the label has to say whose judgment it is: which checker, run when, on what version of the source.
This course’s own position, and it is a position rather than a finding: a support badge with no named checker behind it is worse than no badge, because it converts an unverified citation into a verified one in the reader’s mind at zero evidentiary cost. That is the percentage-bar mistake from the next lesson, wearing a tick instead of a number.
Where people get burned
Never draw a green tick, a “verified” badge or a shield icon on a citation nobody or nothing has checked. This is the site rule about a liveness marker nobody earned, applied to the one surface where the cost of getting it wrong is highest.
The failure mode is specific and it compounds. Once a reviewer learns that the badge appears on every citation, the badge stops carrying information, but it does not stop carrying reassurance. They read faster, click less, and approve more, on a signal that was never measuring anything.
Retrieval check
A stakeholder asks you to show “51.5% of citations are unsupported” next to your component, citing the Stanford paper. What is wrong with that, in three ways?
Check your answer
The number is the wrong one. 51.5% is the share of generated sentences fully supported by their citations, which is recall. The precision figure — the share of citations that support their associated sentence — is 74.5%. Reporting one as the other conflates two failures the paper deliberately measures separately.
It is not about your system. The measurement covers Bing Chat, NeevaAI, Perplexity.ai and YouChat, evaluated for a paper submitted in April 2023. Printing it beside your own component presents someone else’s systems as a property of yours.
It is stale by construction. These are production products that have shipped continuously since. The paper does not license any claim about their behaviour now, let alone about models released since.
What the paper does license is the design argument: unsupported citations occur at rates high enough that a review surface must treat them as a normal case. That belongs in the lesson, in the documentation and in the interview answer. It does not belong on the component as a statistic.
Check your recall
Answer from memory — no scrolling back.
Hands on
Give the anchor a state model and a way to disagree
Done when: ARTIFACT.md’s “Unsupported-citation handling” field names the states an anchor can be in, what each one renders, which of them your run can actually produce today, and what a reviewer’s disagreement records.
- Write the states. Start with cited-and-unchecked, checked-and- supported, checked-and-contradicted, and add any your domain needs. Beside each, the visual treatment in one phrase.
- Mark which states your pipeline can produce right now. If the honest answer is only the first, the component has one state and the other two are specification. Write that down rather than building all three and leaving two unreachable.
- Take one citation from your captured run and check it by hand. Open the source, read it, decide whether it supports the sentence. Time yourself. That number is the cost you are designing down, and it is usually longer than people guess.
- Design the disagree control: where it sits, what it captures, and where the record goes. It must attach to the citation rather than the response, and it must carry the sentence, the source and the passage with it.
- Write the rule for automated checks, whether or not you run one today. Which checker, what the label says, and what the badge does when the source has changed since the check.
- Grep your own mockups and component for a tick, a shield or the word verified. Anywhere one appears without a checker behind it, delete it or attribute it, then note in
ARTIFACT.mdwhich you did.
What this does not cover
This lesson was about whether the evidence backs the claim. It said nothing about how sure the system is, which is a separate number arriving by a separate mechanism and carrying a separate set of lies. The confidence lesson closes this module by taking apart what a model actually means when it says it is 87% sure, and what a component is allowed to render in place of a percentage bar.
It also left the reviewer’s disagreement at the point of capture. What a team does with a stream of labelled precision failures — evaluation harnesses, scoring, judge pipelines — is deliberately out of scope for this whole course, which renders agent work rather than grading it.
Read this next — primary source
Evaluating Verifiability in Generative Search EnginesNelson F. Liu, Tianyi Zhang, Percy Liang (Stanford), Findings of EMNLP 2023; arXiv:2304.09848, submitted April 2023, revised October 2023 — free
Read the methodology, not just the headline numbers. The paper builds a human evaluation of four production systems and has to define, precisely, what it means for a sentence to be supported and for a citation to support a sentence — and those two definitions are the vocabulary the rest of this lesson runs on. Also read the section on the trade-off they observe between fluency and verifiability, which is the uncomfortable part: the responses people rated highest were not the ones best supported by their citations. That is the whole design problem in one result, and it is why a review surface cannot take a well-written answer as evidence of a well-sourced one.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.