PII in a trace view
A trace is built to show every input and output of every step, so a trace UI is a personal-data surface by construction, and the redaction question has to be answered before the view is useful rather than after.
A field came back wrong. The extraction agent read a policy number off page four and put it in the wrong slot, so you open the trace to find out where it went sideways. The trace shows the system prompt, the retrieved chunks, the tool call arguments, the intermediate reasoning, the completion. You find the bug in about ninety seconds. The view is excellent.
It is also, on that screen, a full copy of a stranger’s mortgage paperwork, sitting in a third-party product, viewable by anyone with a company login. That is not a bug in the trace tool. It is what a trace is.
The default is on, and OWASP names the products
The claim “observability tools log everything by default” gets repeated a lot without a source. There is one, in the 2026 edition of the OWASP GenAI LLM Top 10, and it names products rather than generalising:
“Observability platforms (Langfuse, LangSmith, Datadog LLM Observability) log full prompts, completions, chunks, and traces by default.” (OWASP, LLM02:2026)
Use the named list, not a generalisation. The moment you say “all observability tools” you have made a claim OWASP did not make and that the session-replay category, from the review-surface lesson, already contradicts — Sentry masks everything by default. Say the three products OWASP said, and check the fourth one yourself.
Get the identifier right, too. This is LLM02:2026, in the edition published in August 2026. The list was reshuffled between editions and several entries moved several places, so an identifier without a year attached is a coin flip. It is also the single easiest way to lose a room: quote a stale number to somebody who reads this list for a living and everything after it gets discounted.
Why the surface is bigger than the answer
The instinct is to think about the completion, because the completion is what the user sees. The same OWASP entry says the disclosure surface is much wider, and this sentence is the one to be able to recite:
“…disclosure surfaces extend beyond final answers to include tool-call arguments, reasoning traces, retrieved chunks, multimodal output, logs, telemetry, embeddings, and observable inference properties.” (OWASP, LLM02:2026)
Map that onto the extraction agent and the list stops being abstract. Tool-call arguments carry the document identifier and the storage path. Retrieved chunks are the document. Multimodal output is the page image the vision model was handed. Reasoning traces are the model narrating, in plain language, what it thinks it is reading off somebody’s payslip. Every one of those is a node in the tree your trace UI draws.
What the vendor documentation confirms, and what it gives away
LangSmith is both a product OWASP names and a company documenting its own product, so read it with that doubled interest. Its description of a trace is a description of the problem:
“a hierarchical tree representing the full execution showing the exact inputs, outputs, and metrics for each node” (LangChain, LangSmith documentation — vendor)
The same page states that LangSmith’s own terms prohibit PII from being logged to the service. Hold those two facts together. The product is built to show exact inputs and outputs, and the terms of service say do not put personal data in it. Reconciling those is the customer’s job, and the page exists to help.
The detail worth carrying into a meeting is the shape of that help. LangSmith documents three separate mechanisms for keeping data out: an environment variable that hides inputs and outputs wholesale, custom masking functions applied client-side, and regex-based anonymisers. A vendor ships three overlapping controls for one problem when no single one of them is complete. That is not a criticism of LangSmith; it is evidence that “we redact PII” is a sentence that needs a follow-up question about which mechanism, applied where, and what it misses.
Where people get burned
Do not present redaction as the answer, and do not let anyone else present it as one. OWASP’s own mitigation is not a redaction technique. It is “Classify and redact reasoning traces as first-class output” followed by “Never log raw traces to unrestricted observability” — the second half is an architectural control about who can reach the backend, and it is there because the first half does not close the gap on its own. A regex catches the identifier formats somebody thought of. It does not catch a name in an OCR block or a diagnosis in a retrieved chunk.
What OpenTelemetry does and does not tell you
Several of these vendors are converging on OpenTelemetry’s semantic conventions for GenAI spans, and it is tempting to reach for a vendor-neutral standard as the source for “on by default.” It will not carry that weight. What the conventions actually require is an off-switch:
“instrumentations that support it MUST offer the ability to turn off capture of prompts and completions” (OpenTelemetry, semantic conventions for GenAI spans)
That establishes that off is possible. It does not establish what any given instrumentation ships as its default, which is a per-vendor fact you have to check per vendor. The same document does carry a warning worth quoting to a platform team, because it puts the user at the centre of it: end users of these applications “may input sensitive information or personally identifiable information (PII) that they do not wish to be sent to a telemetry backend.”
Two sources, two different jobs. OWASP is where the “by default” claim comes from. OpenTelemetry is where the mechanism and the warning come from. Attributing either one to the other is the kind of error that a platform engineer will catch, and it costs you the flag.
The part that is actually yours
Almost everything above belongs to whoever owns the agent runtime and the observability contract. Three things do not.
- The internal trace viewer, if you build one. A product team that outgrows the vendor UI builds its own, inside the admin app, and it inherits the admin app’s access model. That model is frequently “anyone who can log in.” A screen rendering exact inputs and outputs behind a role that was designed for a settings page is a front-end decision.
- What the support surface shows. The moment a trace excerpt is surfaced in a customer-facing or partner-facing screen, it stops being an internal debugging artefact and becomes a rendered surface with the properties the first module described.
- Whether the redaction happens before the network call. Client-side masking applied in the browser and server-side scrubbing applied at the collector are different guarantees with different failure modes. Knowing which one is in place is a question, not a control you write.
Check your recall
Answer from memory — no scrolling back.
Retrieval check
Why is “the trace view is internal-only” a weaker answer than it sounds?
Check your answer
Because internal-only describes who is allowed in, not how many people that is or what they can do once inside. A company-wide SSO login is internal. A support contractor with an admin seat is internal. An offboarded employee whose access was not revoked was internal for the whole window. And the trace backend is usually a third-party product, so “internal” already means the data left your systems to get there.
The version that has an answer is narrower: which role can open this view, how many holders does that role currently have, and is opening a trace an event that is itself logged. Those three questions get numbers back. “Is it internal?” gets a yes, and a yes closes the conversation without changing anything.
Hands on
Write the trace-view flag
Done when: A one-page flag naming: the trace tooling in use and its documented default with a URL, which of OWASP’s listed disclosure surfaces are populated with personal data in your own agent, whether redaction happens client-side or at ingest, and which role can open the view. Every cell is either a checked fact with a source or the words “not established”. No mitigation is proposed as sufficient.
- Name the tooling. Not the category — the product and the version. If more than one thing receives spans, list them all; a separate error reporter frequently carries the same payloads.
- Open one real trace from a non-production environment, or a production trace if that is what you have access to and it is appropriate for you to look at. Walk OWASP’s list against it: tool-call arguments, reasoning traces, retrieved chunks, multimodal output, logs, telemetry, embeddings. Mark each one populated, empty, or not applicable.
- For every populated row, write what kind of personal data is in it in the plainest words available. “The retrieved chunk is the full text of a customer bank statement” travels further in a meeting than “chunks may contain PII.”
- Find where redaction happens, if it happens. In the browser, in the SDK before the network call, or at the collector after it. Write which, and write what you could not determine.
- Establish who can open the view. Ask for the role name and the count of people currently holding it. A count you were told is still better than a count you assumed, as long as you record which it is.
- Write the What I am not claiming line before you write anything else in the flag. Start it with: this is not a claim that the tooling is misconfigured, and it is not a claim that any data has been accessed improperly. Then bring the page in and I will push on any row that says “probably”.
What this does not cover
This lesson stops at what the trace contains and who can see it. It has said nothing about where the trace backend physically is, and that is the next question a privacy owner will ask you: a trace containing a European customer’s document, sitting in a vendor’s infrastructure somewhere else, is a transfer. The residency lesson gives you the two facts that turn that unease into a question with an answer — whose data it is, and which border it crosses — and it is careful about what the regulation does and does not require.
It has also said nothing about how long the trace lives. Trace retention, transcript retention and model-vendor log retention are three different systems with three different defaults and three different owners, and conflating them is how a confident answer turns out to be about the wrong database. That is the retention lesson.
Read this next — primary source
LLM02:2026 Sensitive Information DisclosureOWASP GenAI Security Project, OWASP GenAI LLM Top 10 2026 — free, published August 2026, fetched 2026-09-05. Text lives as Markdown in the project repository; the 2026 edition has no per-entry web pages yet, and the obvious genai.owasp.org URL still serves the 2025 list.
This lesson takes two sentences: the one naming observability platforms that log full traces by default, and the one enumerating the disclosure surfaces beyond the final answer. The rest of the entry is a mitigation catalogue tiered by effort, and it is worth reading in full for a reason that is not about you implementing any of it — the tiers tell you which mitigations a platform team considers cheap, which is what determines whether your flag gets a fix this quarter or a roadmap item.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.