The wait is the interface

The points-and-miles chatbot answers a real question: what is the cheapest way to get two people from Austin to Tokyo in business class in March, using the points I already have? Answering it properly means checking award availability across a few programs, working out which transfer partners reach them, and doing the arithmetic on what is actually in the account. It is a genuinely good answer when it arrives.

It arrives all at once, after a wait, as a wall of text. During that wait the interface offers one thing: a spinner. The user cannot tell whether it is thinking, stuck, or already finished and rendering. They cannot stop it. If they switch tabs and come back, they get whatever the browser happened to preserve. If it fails at 80 percent, they get nothing and start over.

None of that is a bug. It is what request, wait, render looks like when the wait grows from 300 milliseconds to forty seconds. The architecture did not change. The product did.

Three numbers, and the point where they run out

The response-time thresholds everyone half-remembers come from Jakob Nielsen, and they are older than most of the web: 0.1 second is “the limit for having the user feel that the system is reacting instantaneously”; 1.0 second is “the limit for the user’s flow of thought to stay uninterrupted”; 10 seconds is “the limit for keeping the user’s attention focused on the dialogue” (Nielsen, 1993). NN/g later split the middle band into a rendering rule: a looped animation for 2–9 seconds, a percent-done indicator for 10 or more (Sherwin, 2014). Nielsen Norman Group sells UX training and research reports, so this is a consultancy publishing its own foundational work — but it is the primary source for these numbers, not a summary of someone else’s.

Read the ten-second rule carefully, because it does not say “users get impatient.” It says attention leaves. Nielsen is explicit about the consequence: past ten seconds “users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done”. That is not a styling note. It is a statement that the user will physically go somewhere else, and the interface has to survive their absence.

Now notice where the guidance stops. The prescribed fix above ten seconds is a percent-done indicator — and a percent-done indicator requires knowing the denominator. A file upload knows its byte count. A report generator knows its row count. An agent answering the Tokyo question does not know how many tokens the answer will be, how many tools it will call, or whether the third tool call will come back empty and send it down a different path. There is no honest denominator.

Streaming does not make it faster

This is worth stating flatly because the pitch usually implies otherwise. Streaming the Tokyo answer does not reduce the time until the answer is complete. If anything it adds a small amount of overhead. What it changes is when the first piece arrives — and that one change cascades into a different application.

First content at 900 milliseconds instead of forty seconds buys three things that no spinner provides. It proves the system is alive, continuously, rather than asserting it once. It gives the user something to do during the wait — reading is a task, waiting is not — which is the mechanism by which streaming beats the ten-second attention limit rather than merely decorating it. And it lets the user judge direction early: two sentences in, they can tell the agent misread “business class” as “business trip” and kill it, instead of discovering that after forty seconds of generation they already paid for.

There is also a blunt infrastructural reason, and it comes from the model vendors themselves. Anthropic — documenting its own API, so read it as such — warns against long non-streaming requests on the grounds that “some networks may drop idle connections after a variable period of time, which can cause the request to fail or time out without receiving a response,” and states that its SDKs “validate that your non-streaming Messages API requests are not expected to exceed a 10-minute timeout”. A long request that produces nothing on the wire looks identical to a dead connection, to every intermediary between you and the model. A stream is, among other things, proof of liveness that middleboxes can see.

And one detail from that same page that will matter for the rest of this course: “an error can occur after the API returns a 200 response. In that case, error handling doesn’t follow these standard mechanisms”. Once you stream, the HTTP status code stops being where failure lives. Your error handling moves inside the response body. That is not a streaming inconvenience; it is a structural change to how the client knows anything went wrong.

What the long wait forces you to build

Here is the concrete inventory. Each of these is a control the two-second version of the chatbot never needed, and each one is absent from the current build:

• Continuous proof of life. Not “a request is in flight” asserted once at submit, but evidence arriving on an ongoing basis. A spinner cannot distinguish working from hung; a stream can, because it stops.
• Narration of what it is doing. “Checking award space on ANA” is a different claim from “loading,” and it is the claim that makes a forty-second wait tolerable. This is only possible because an agent stream carries more than text.
• An exit. Nielsen’s ten-second rule says attention will leave. If it leaves and cannot come back — or leaves and the work keeps billing — that is a design failure, not user error.
• Survivable absence. The user will switch tabs. The question is what they find on return, and today the honest answer for most streaming builds is “whatever the browser happened to keep.”
• A way to tell finished from still-going. When text appears incrementally, a pause is ambiguous: thinking, calling a tool, rate-limited, or done. The end of a stream must be rendered as an event, not inferred from silence.
• In-band failure. Because the status code is already 200, the UI needs a rendering for “this answer stopped being produced” that is distinct from both “done” and “still coming.”

Notice that only the first two are about making the wait pleasant. The other four are about the wait failing — which is the half that every streaming demo skips, and the half that decides whether a surface can be handed to somebody else’s product team.

What this does not cover

This lesson argues that the surface has to stream and names what streaming obliges you to build. It says nothing about how the bytes actually get to the browser — and that choice is not a detail, because the transport you pick on localhost is chosen for reasons that stop applying the moment a buffering reverse proxy sits in front of it. The transport lesson takes that on directly: SSE, WebSocket, chunked HTTP and polling, scored on how each one behaves in somebody else’s infrastructure rather than on yours.

The narration item in the affordance list is also a promise this lesson cannot keep on its own. Saying “checking award space on ANA” requires the stream to carry tool calls as first-class events, not text — which is what the agent-stream lesson at the end of this module builds, and what the raw-events lesson before it makes visible.