Streaming LLM responses feels like it should be a one-liner. In practice you hit runtime constraints, client hydration quirks, and timeout defaults that were never designed for 30-second model calls. I've shipped this twice — once in CodexGenAI and once in HireOS — and the second time went much faster. Here's the version that holds up.
Route handlers vs Server Actions for streaming
The stable choice today is a route handler that returns a Response wrapping a ReadableStream. Route handlers give you direct control over the response object, headers, and the stream itself — which is exactly what streaming needs.
Server Actions are excellent for mutations and progressive enhancement, but they aren't the right primitive for streaming a token feed to the client; the ergonomics fight you. Use a route handler (app/api/chat/route.ts) for the stream, and reserve Server Actions for the surrounding mutations. Decide the runtime explicitly too: the edge runtime streams well and has low cold starts, while the Node runtime gives you the full API surface — pick per your dependencies.
ReadableStream and SSE plumbing
The shape is: create a ReadableStream, read from the provider's streaming response, encode each chunk, and enqueue it. Wrapping chunks in the Server-Sent Events envelope (data: ...\n\n) gives the client a clean, well-understood format and plays nicely with how browsers handle streamed text.
A TransformStream is useful when you need to reshape provider chunks into your own envelope on the way through — for instance normalising different providers' streaming formats into one the client always sees. The detail people miss: make sure chunks are actually flushed rather than buffered, especially on the edge, or the whole "streaming" effect collapses into one delayed dump.
Backpressure and cancellation
This is the part that quietly costs money if you ignore it. When a user navigates away or closes the tab mid-stream, you must propagate that cancellation upstream and abort the provider request — otherwise the model keeps generating, you keep getting billed, and the connection dangles.
Wire the request's AbortSignal through to the upstream fetch to the provider, and handle the stream's cancel() so an abandoned client tears down the whole chain. Treat "user left" as a first-class path, not an edge case.
Rendering tokens on the client without jank
The naive approach — appending each token to useState and re-rendering — works until the response is long, then every keystroke-sized chunk triggers a full re-render and the UI stutters.
What works better:
- Accumulate into a ref and flush to state on an interval or animation frame, rather than setting state on every single chunk.
- Keep the streamed text in its own small component so re-renders don't cascade through the rest of the page.
- For the cursor/typing effect, animate with CSS rather than re-rendering layout, to avoid thrash.
The goal is smooth perceived output, which is a rendering-cadence problem, not a network one.
Error and timeout handling at the edge
Long model calls collide with platform timeouts. Cloudflare Workers have CPU-time limits; Vercel functions have their own timeouts — both shorter than a worst-case generation if you're not careful. Design for partial output: if the stream dies mid-response, surface what you have plus a clear error, rather than discarding everything.
Also distinguish the two failure types, because they need different UX: a network/transport error (retry the request) versus an LLM error mid-stream (a provider 429, a content filter, a context-length error — show the specific cause). Conflating them leads to "something went wrong" messages that help no one.
FAQ
Route handler or Server Action for streaming?
Route handler returning a ReadableStream. Server Actions aren't built for token streaming today.
How do I stop getting billed when a user leaves mid-stream?
Propagate cancellation through an AbortSignal to the upstream provider fetch, and handle the stream's cancel() to tear down the chain.
Why does my stream arrive all at once instead of token by token? Chunks are being buffered, not flushed — common on the edge. Ensure each chunk is enqueued and flushed as it arrives.