What must never be logged
A review gate sits on top of exactly the content that must not leave it — the discipline is emitting the shape of the correction without the payload, and knowing which regulator’s principle you are answering to.
The most useful event you could design for HouseWarm’s gate looks like this:
field_corrected {
field: "borrower_name",
from: "Jhon Smyth",
to: "John Smith"
}With that event you could cluster corrections by kind, find the fields the agent gets wrong in a systematic way, measure whether a correction was a typo fix or a whole-value replacement, and build the eval set the evals module is about. It is the single highest-value row in the schema.
It also takes a named individual out of a mortgage document and puts them in an analytics warehouse, replicated to whichever region the vendor uses, retained on whatever schedule the vendor defaults to, queryable by anyone with a dashboard login. A review gate sits on top of exactly the content that must not leave it. That is not an unlucky coincidence. It is what a review gate is: a human looking at the sensitive thing to check whether the machine read it correctly.
The discipline: emit the shape, not the payload
The instinct when told to drop a property is to drop the event. That is the wrong trade, and it is unnecessary. Almost everything you wanted from from and to survives a transformation into a shape that carries no content:
| Wanted | What actually gets emitted |
|---|---|
| The old and new values | changed: true, a bucketed edit distance, and a change_kind enum — casing, punctuation, digits, partial replacement, whole-value replacement |
| Which field, so you can find systematic failures | The schema key, borrower_name — a name from your own extraction schema, not a value from anyone’s document |
| How confident the agent was | confidence_bucket, an enum, because a float to two decimal places is a near-unique fingerprint across eleven fields |
| The raw OCR text the reviewer consulted | ocr_opened: true and a bucketed character count |
| Why the reviewer escalated | A fixed enum of reasons. Never a free-text box, because a free-text box is an unbounded content channel with a friendly label |
| Which document, so events can be joined into a session | A per-session pseudonymous id that does not resolve to the document anywhere the analytics store can reach |
Read the right-hand column against the metrics. Correction rate still computes. Acceptance rate still computes. The clustering you wanted — which fields fail, and how — still works, because change_kind is the part of the correction that generalises and the borrower’s name is the part that does not. The eval-set argument in the evals module is the one real casualty, and it is solved by keeping the labelled pairs inside the system that already holds the document rather than by loosening this table.
The principle, and whose it is
The authority here is the European Data Protection Board’s guidance on Article 25 of the GDPR, data protection by design and by default. Its core requirement in section 3.5 is that only personal data that is adequate, relevant and limited to what is necessary for the purpose shall be processed. Four adjectives, each of which is a question you can put to a property in a schema row.
The part that turns it into a design procedure is the verification duty that follows. Controllers are asked to check whether the purpose could be achieved
“by processing less personal data, or having less detailed or aggregated personal data or without having to process personal data at all”
That is the never-logged table written as an instruction. Less detailed is confidence_bucket instead of the float. Aggregated is the bucketed edit distance instead of the strings. Without processing personal data at all is change_kind. The guidance also lists the design elements it expects to see — data avoidance, limitation, relevance, necessity, aggregation, pseudonymisation, anonymisation and deletion — and asks separately that the degree of identification be minimised, which is the sentence that rules out the two-decimal confidence score even though a confidence score is not obviously personal data at all.
The worked example is the one to carry into a design review. The guidelines describe a transport operator that deliberately does not store the ticket identifier alongside the data it collects, because storing it would let the records be linked back to an individual and the operator’s purpose does not require that link. A field it could trivially have kept, not kept, on purpose, with the reason written down. That is the shape of every row in section 3 of your spec.
What this lesson is not
It names a principle and a regulator. It does not tell you what the law requires of your product, and the mission puts legal advice out of scope for good reason: the answer depends on your data, your jurisdiction, your processors and your contracts, none of which a course can see.
Two honest limits on the sourcing, as well. The EU regime is the one with an accessible, official, free design document, and that is why it is here — not because it is the only regime that applies to you. The UK Information Commissioner’s own data-minimisation pages returned HTTP 403 on every attempt made for this course, so no ICO claim appears anywhere in it. The obvious US-side counterpart, the NIST Privacy Framework, downloaded but its text could not be extracted, so it is not cited either. If you need a US authority for a never-logged table, that gap is real and you will have to close it yourself.
The standards bodies already made this call one layer down
You are not the first person to notice that the interesting telemetry is the sensitive telemetry. OpenTelemetry’s work on conventions for generative-AI systems arrives at the same default, from the model side: prompt and response content is not captured unless someone explicitly turns it on. The project’s own write-up states that no prompt content or tool arguments are captured with GenAI telemetry, as these can contain sensitive data, and the event that does carry message content is marked opt-in rather than recommended. That is the OpenTelemetry project describing its own conventions, so read it as a design decision by the people who made it rather than as independent validation — but the decision is the same one, taken by people with the same problem.
One caveat that the routing lesson goes into properly: those conventions are not a settled standard. Every generative-AI attribute, span, metric and event in the OpenTelemetry registry currently carries a stability status of “Development,” and names in that namespace have already changed once. Take the content-off default as a well-reasoned default. Do not take any specific attribute name as fixed.
The transferable point is that content-off is the sane starting position and each exception is argued for individually. A review gate is the same problem one layer up: the model saw the document, the reviewer saw the document, and the telemetry about both should be able to describe what happened without reproducing what was seen.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Someone proposes a free-text “reason for escalation” field, arguing that reviewers will write process notes, not document content. Why is that argument wrong regardless of what reviewers intend?
Check your answer
Because the field’s contents are not bounded by anyone’s intention. A reviewer escalating an unreadable name will type the name — not out of carelessness, but because naming the thing is how a person explains a problem. The property is an unbounded content channel with a helpful label, and it fails the necessity test in the guidance on its own terms: the purpose is knowing why escalations happen, and that purpose is achievable with less data.
The replacement is a fixed enum, built from the reasons that actually occur, with the list revisited when reviewers pick “other” too often. That costs you a design conversation about what the categories are, which is a conversation worth having anyway. If free text is genuinely necessary, it belongs in the system that already holds the document and is governed accordingly — never in the analytics event.
Hands on
Fill section 3, then run it against section 1
Done when: Section 3 of INSTRUMENTATION.md has one row per excluded field with its authority and its replacement, every property in section 1 has been checked against it, and any property that failed has been rewritten as a derived shape rather than deleted along with its event.
- Open
learning/agent-evaluation/INSTRUMENTATION.mdat section 3. Write one row per excluded thing on HouseWarm’s gate: field values, raw OCR text, the source-image crop or any reference to it, free-text reasons, precise confidences, and any document identifier that resolves outside the session. - For each, fill the authority cited column with the specific principle rather than the word “GDPR.” Necessity, relevance, aggregation, degree of identification — name the one that actually does the work for that row.
- Fill the what is emitted instead column for every row. A row with an empty replacement cell is a row that will be argued away in the first meeting where someone needs the data.
- Now go back to section 1 and check every property against the table. Where one fails, rewrite the property. Deleting the whole event is almost always the wrong repair and is how a schema quietly loses the revert.
- Add one line to section 4 naming the privacy reviewer who has to sign this off, and the one question you most need them to answer. If you do not know who that is at HouseWarm, that is itself the open question.
- Bring section 3 into the chat. I will argue for logging the old and new values, from the strongest position — that it would make the eval set possible — and your job is to answer it without giving ground on the payload.
What this does not cover
Section 3 constrains what a payload may contain. It has nothing to say about where the payload goes, and the destination changes the answer: the same interaction can be safe to record in one system and reckless in another, and the two systems have different audiences, retentions and access models.
The routing lesson fills the column the schema lesson told you to leave blank — analytics, the trace, or both — and shows the specific failure that column exists to prevent, where an event lands only in a sampled, short-retention store and a metric becomes uncomputable without anyone noticing.
Read this next — primary source
Guidelines 4/2019 on Article 25 — Data Protection by Design and by Default (version 2.0)European Data Protection Board — free, official regulator guidance, adopted after public consultation
Section 3.5, paragraphs 73 to 76, is four pages and is the whole authority behind the never-logged table. It is worth reading in full because it is written as design guidance rather than as a prohibition list: it gives a verification duty you can run against any proposed event property, a set of named design elements that map almost one-to-one onto payload decisions, and a worked example of an operator deliberately not storing an identifier it could easily have stored. Where it stops: this is the EU regime and nothing else, it is guidance rather than statute, and it says nothing about your product. The course names the principle and the regulator. It does not assess anyone’s exposure, and neither should you.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.