Analytics, or trace
The same interaction belongs in two systems for two audiences, and putting an event in the wrong one is how a metric becomes uncomputable without anyone noticing.
Every event in section 1 now has a name, a trigger, a property list and a metric. One column is still empty, and it is the one that decides whether any of the work survives contact with a query.
Two systems are waiting for these events and they are not interchangeable. Product analytics is aggregate: many sessions, many users, long retention, queried months later by somebody who never saw the code, in service of a question about a population. Tracing is per-request: one run, high cardinality, rich detail, short retention, opened by an engineer who is trying to understand why this specific document came back the way it did.
Put an event in the wrong one and nothing breaks. That is the problem. The event is emitted, the dashboard is built, the number appears, and the number is computed from a store that was sampling.
Route by the question, then check the store
The rule is short, and it is this course’s own framing of a split that is otherwise uncontroversial engineering practice. If the question is about a population, it is analytics. If the question is about one run, it is the trace.
- “What share of documents were corrected last month?” is a population question. Analytics.
- “Why did field seven on this document come back at low confidence?” is a one-run question. Trace.
- “How long do reviewers dwell before approving?” is a population question, and the answer is a distribution. Analytics.
- “What did the reviewer actually see on the screen when they escalated that one?” is a one-run question, and it is the one that gets asked in an incident. Trace.
Then check the store before you commit a metric to it. Ask the team who owns the trace backend two questions: what is the sampling rate, and what is the retention window. If the sampling rate is anything other than none, a count from that store is a count of an unknown fraction. If the retention window is shorter than the metric’s window, the metric silently truncates at the boundary and the truncation looks like a trend. A metric computed off a sampled, short-retention store is not a metric; it is a sample with a confident label. Neither of those properties is universal, which is exactly why you ask rather than assume.
Some events belong in both, with different payloads
The approve is the clearest case, and it is the one that shows why “both” is a real answer rather than a hedge.
The analytics copy is bucketed and anonymous: document_reviewed with an outcome from the four-way partition, a corrected-field count, a dwell bucket, an evidence-opened boolean. It has to survive a year, join across sessions, and reveal nothing about a document. The trace copy is the opposite: a span on the review carrying the run identifier, so an engineer holding a complaint about one extraction can walk from the reviewer’s decision back through the model call that produced it.
Same interaction, two payloads, two retentions, two audiences. Write both in the destination cell, and write what differs, because “ both” without the difference will be implemented as the same payload sent twice, which is the never-logged table defeated by an ambiguity in a spreadsheet.
The emerging convention on the trace side, and how green it is
There is a standards effort for the trace half of this, and it is worth knowing about, but the first thing to say about it is its maturity.
OpenTelemetry’s semantic conventions for generative AI are not stable. Every attribute, span, metric and event in the gen_ai namespace carries a stability status of “Development” rather than “Stable.” With the v1.42.0 release on 12 June 2026 the conventions moved out of the main semantic-conventions repository into a dedicated repository of their own, which as of the research date for this course has no tagged release and whose documents still read “Status: Development.” Names in the namespace have already changed at least once: gen_ai.system became gen_ai.provider.name, and gen_ai.usage.prompt_tokens became gen_ai.usage.input_tokens.
A project rarely states its own immaturity as bluntly as an outside reader will, which is why the plainest statement of the position comes from a third-party audit rather than from the docs: Development status explicitly means names can still change, and the practical recommendation that follows is to treat the wire protocol as your integration point rather than betting a schema on particular attribute names. That is the same shape of caveat this course attaches to the AI Act reading copy: use the source, name what it is, and say the sentence out loud every time rather than once in a footnote.
Say the status, every time
If you name these conventions in an interview, in a spec, or in a design review, name the status in the same breath. “We align the trace to OpenTelemetry’s generative-AI conventions, which are still at Development status and have been renamed once, so we treat the shape as the commitment and the names as changeable.”
That sentence costs nothing and is the difference between sounding current and being current. A spec that presents these names as fixed infrastructure will read as dated within a release or two of shipping, and the person who notices will be the engineer who has to rename everything.
What the conventions are actually useful for here
Two things, both about shape rather than names.
First, they model an evaluation result as telemetry. There is a defined event, gen_ai.evaluation.result, at Recommended requirement level and Development status, which captures the result of evaluating generative-AI output for quality, accuracy or other characteristics, carrying an evaluation name, a score value or label, and an explanation. That is a direct answer to a routing question the evals module will raise: a judge score is a per-run fact about one output, so it belongs in the trace next to the run it scored, not in product analytics where it would be averaged into a number nobody can trace back to anything.
Second, they put content where this course would put it. The event that carries message content — gen_ai.client.inference.operation.details, holding input messages, output messages, system instructions and tool definitions — is opt-in rather than recommended, and is defined as a separate record kept apart from the spans. The conventions independently arrive at the routing rule the never-logged lesson argued for: the payload, if it is captured at all, lives in the short-retention, access-controlled, per-run store, and never in the thing that gets averaged. The published span hierarchy underneath all this is worth a look too — an agent invocation wrapping the model calls and tool executions it made — because it is the structure your review-gate span has to attach to.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Why is “both” only a real answer when the destination cell also says what differs between the two payloads?
Check your answer
Because the default implementation of “both” is one payload emitted twice, and that payload will be the richer one. The trace copy carries a run identifier so an engineer can find the extraction. Send that identifier to analytics as well and you have put a durable join key into the long-retention store, which is the exact thing the never-logged table pseudonymised the session id to prevent.
So the cell reads: both — analytics gets outcome, corrected count, dwell bucket, evidence-opened; the trace gets the run id and the per-field detail. Two payloads, written down, so the difference is a spec decision rather than an engineer’s guess at three in the afternoon.
Hands on
Fill the destination column, and defend every cell
Done when: Every row in section 1 of INSTRUMENTATION.md has a destination with a one-line reason, every “both” row states which payload goes where, and section 4 records the sampling rate and retention window of the trace store — or the fact that you do not yet know them and who does.
- Go through section 1 row by row. For each, write the question the event answers as either a population question or a one-run question, then let the destination follow from that rather than from habit.
- For any row you mark both, split the property list in the cell. Analytics payload on one side, trace payload on the other. If they come out identical, one of the two destinations is wrong.
- Write down, in section 4, the two questions for whoever owns the trace backend: sampling rate and retention window. If you already know the answers, record them and mark any metric they make uncomputable. If you do not, that is an open question with a named owner, which is what section 4 is for.
- Add one row for the review-gate span itself — the trace-side record of a human decision, carrying the run identifier and nothing from the document. Note next to it that the surrounding span hierarchy you would attach to comes from a convention still at Development status.
- Reread section 3 with the destination column filled. Any property that was acceptable when you thought it was going to a per-run store, and is not acceptable in a store retained for a year, has to change now.
- Bring the finished section 1 into the chat. I will pick the three rows whose destination reason is weakest and argue for the opposite routing.
What this does not cover
Section 1 of the spec is now complete: every event named, triggered, typed, constrained and routed. It is a UI instrumentation spec, and it is finishable by you alone, which is what makes it the right artifact for this role.
What it cannot do is tell you whether the agent is any good. That question belongs to the eval stack, and the mission scopes owning that stack out on purpose — but you still have to argue with the person who owns it, and the argument turns on knowing what their tooling is structurally unable to see. The evals module opens there, with what an offline eval scores and what it is silent about, and then works through the failure modes of a model grading a model, an evidenced preference among three tools, and the loop that turns corrections captured at your gate into the evaluation set.
Read this next — primary source
OpenTelemetry semantic conventions for generative AI — eventsopen-telemetry/semantic-conventions-genai — free, official project text. Every element in it currently carries stability status “Development,” and the repository has no tagged release
Read it for the shape of the decision rather than for the names. It shows a standards body working out what belongs in a trace about an AI system, what has to be opt-in because it carries content, and how an evaluation result is modelled as telemetry rather than as a report — which is the exact routing question this lesson asks about a review gate. Where it stops, and this matters: it is at Development status, not stable, and the conventions have already been through at least one renaming pass. Read it to learn what the trace is for. Do not build a schema that hard-codes its attribute names and assume they will still be there.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.