Streaming responses without breaking your UI
Streaming tokens as they arrive transforms perceived latency, but partial output is often invalid — half a markdown fence, an unterminated JSON object — and the stream can fail mid-response. Robust implementations render defensively, buffer structured output, and always render a terminal state.
A response that takes eight seconds to arrive complete feels slow. The same response streaming from the first token feels immediate, even though it finishes at the same moment.
That is a real gain and it comes with real complications, most of which stem from one fact: you are now rendering text that is not finished.
Partial output is invalid output
Mid-stream you will hold a markdown code fence that has opened and not closed, a list item with no terminator, a table with one row of a header. Passing that to a strict renderer produces flicker, broken layout or an exception.
Render defensively: either a renderer tolerant of incompleteness, or display the raw text until a completion boundary is reached. The naive approach — full markdown rendering on every chunk — is what causes the jumping layout people associate with streaming.
Structured output is different
JSON has no useful partial state. Show a progress indicator by all means, but buffer until the stream completes, then parse and validate once. Acting on half an object is a bug waiting for a slow network.
Design the failure
Streams drop. A response that stops after two sentences looks exactly like a response that was two sentences long, and the user cannot tell which happened.
Track completion explicitly and mark an interrupted message as interrupted, with a retry. This is the single most common omission in streaming implementations and the one users notice most.
Do not fight the scroll
Auto-scrolling to follow new tokens is right until the user scrolls up to re-read something, at which point yanking them back down is infuriating.
Follow while they are at the bottom; stop the moment they scroll away; offer a "jump to latest" control. It is a small piece of state and it is the difference between a pleasant interface and an annoying one.
Keep the cancel honest
A stop button that only hides the output while tokens keep being generated is billing the user for work they cancelled. Abort the request properly, and stop the clock.
Frequently asked questions
Should structured output be streamed?
Stream it for progress if you like, but do not act on it until it is complete and validated. Half an object is not a smaller object; it is an invalid one.
What happens when the connection drops mid-stream?
You are left with a partial answer and no completion event. Track that explicitly and show it — a message that simply stops looks identical to one that finished, which is the worst possible outcome.
