How to Handle Errors Mid-Stream: Dropped Connections and Partial Replies

// In plain English

You've already learned how streaming works: the API holds the connection open and pushes the model's answer to you a token at a time. This article is about the moment that connection breaks while it's running — after the bytes have already started flowing, with half an answer on the screen.

Picture a friend reading you a long recipe over the phone. Halfway through step 4, the call drops. You're left holding part of a recipe. Three questions hit you at once: did you actually lose the call, or is your friend just pausing to turn the page? Is the half-recipe you wrote down good enough to start cooking, or useless? And if you call back, will they start over from the top, or pick up where they left off — and either way, are you now paying for two phone calls?

A streaming LLM call has exactly those three failure questions. A normal HTTP request gives you one clean signal: a status code at the start, then the whole body. Streaming flips that — the server sends 200 OK the instant it begins, long before it knows whether it will finish. So an error that arrives after the 200 doesn't look like an HTTP error at all. It looks like text that simply stops, or a connection that goes quiet, or a special error message buried inside the stream. Handling those well is the difference between an app that recovers gracefully and one that freezes on a half-answer forever.

// Why it matters

Mid-stream failures are common precisely because streams stay open for a long time — 10, 30, sometimes 60+ seconds for a long answer. That's a wide window for something to go wrong, and the things that go wrong are everyday infrastructure, not exotic bugs:

• The user's network drops. A phone leaves Wi-Fi range, a laptop sleeps, a tunnel swallows a mobile connection mid-answer.
• A proxy or load balancer times out. Many gateways kill any connection that's been "idle" (no new bytes) for N seconds. A model that pauses to think can trip that idle timer even though nothing is actually broken.
• The server restarts or the upstream hiccups. A deploy, an autoscaling event, or a transient provider error lands while tokens are streaming.
• The model itself signals an error mid-stream — the provider can send an error event inside the SSE stream (for example, an overload that hit after generation started).

Why a builder should care: the naive happy-path streaming loop handles none of these. It assumes the loop runs until the model says it's done. When the stream dies early, that loop either hangs forever waiting for bytes that will never come, or it exits silently and shows the user a frozen half-answer with no error and no retry button. Both are worse than a clean failure — the user can't tell whether to wait, refresh, or give up.

And there's a money angle that plain request/response code never has to think about. If you retry a streamed call that had already produced output before it died, you can be billed for the tokens twice — and if that call triggered side effects (wrote to a database, sent a message, called a tool), a blind retry can do those things twice too. Recovering safely means retrying in a way that doesn't double-charge or double-act. That's the idempotency problem, and it's the hard half of this topic.

// How it works

To handle a mid-stream failure you first have to detect it. There are three distinct failure shapes, and they surface differently. Here's the full life-cycle, with the failure points marked.

The three failure shapes

The single most important rule ties them together: track whether you saw the terminal event. Every well-formed stream ends with one — on the Anthropic Claude API it's message_stop (preceded by a message_delta that carries the stop reason and token usage); OpenAI-style streams end with a literal data: [DONE] marker. If your loop exits and you never set a "saw terminal event" flag, the response is incomplete — no matter how much text you collected. That one boolean is your truth source.

Detecting the silent stall with a timeout

A hard disconnect ends the loop, so it's self-announcing. The sneaky one is the idle stall: the connection stays open but no bytes arrive, and a naive for-loop will wait forever. The fix is an inactivity timeout — a timer you reset on every chunk. If it ever fires, you treat the stream as dead and stop waiting.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the env

def stream_answer(prompt: str):
    text = ""
    completed = False  # <- the truth source: did we see the terminal event?
    try:
        with client.messages.stream(
            model="claude-opus-4-8",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        ) as stream:
            for chunk in stream.text_stream:   # yields decoded text deltas
                text += chunk
                yield chunk                    # forward to the UI live
            # We only reach here if the SDK saw the full life-cycle.
            final = stream.get_final_message()
            completed = final.stop_reason is not None
    except anthropic.APIError as e:
        # An error event or a transport failure surfaced as an exception.
        # `text` still holds whatever streamed before the break.
        return {"text": text, "completed": False, "error": str(e)}

    return {"text": text, "completed": completed, "error": None}

// What to show when only part of the answer arrived

Once you've detected the failure and salvaged the partial text, you have a product decision: what does the user see? The wrong answer is "nothing" (a frozen half-answer with no explanation) or "a silent retry that wipes what they were reading." Decide based on what kind of content streamed.

The JSON row is the big trap. Mid-stream, structured output is invalid by definition — you stopped somewhere inside the object. There's no safe way to parse {"name": "Ali into a usable value. So for anything a machine will consume, treat a partial as zero and retry; only free-form text earns the "keep and show" treatment. (This is why the streaming-structured-outputs topic exists at all — see structured outputs explained.)

// Retrying safely without double-billing or double-acting

Now the hard part. You decided to retry — how do you do it without paying twice or triggering side effects twice? The honest truth first: an LLM stream generally cannot be resumed token-perfect. The SSE spec has a Last-Event-ID header for resuming feeds, but model streams don't replay from a byte offset — the provider has no "continue exactly where the socket died" endpoint. So a "retry" almost always means generating again from the start, and that has cost and side-effect consequences you have to manage.

Cap retries and back off

First, don't retry forever and don't retry instantly. A dropped stream is often transient (a network blip), but a persistent failure (a bad prompt, a real outage) will just keep failing. Use a small retry cap with exponential backoff — and only retry failures that are plausibly transient.

This is the same backoff machinery you'd use for a rate-limit / 429 response — but note the difference: a 429 happens before any output, so retrying it is cheap and obviously safe. A mid-stream drop happens after output, so retrying it is the expensive, side-effect-laden case. Same tool, higher stakes.

The double-billing reality

If the model produced 400 output tokens, then the stream died, then you retry and it produces 400 again — you pay for roughly 800 output tokens to get one answer. There's no way around paying for the first, abandoned generation; the provider already did that work. Your levers are: (1) retry as few times as possible (the cap), and (2) capture usage when you can. Token usage arrives in the terminal message_delta event, so a stream that died before that point gives you no usage number for the partial — you simply absorb that cost. Log every retry so a runaway loop shows up in your cost dashboards instead of silently burning budget.

Idempotency: the side-effect half

Double-billing is annoying; double-acting is dangerous. If the streamed call did something — created an order, sent an email, ran a tool — a blind retry can do it again. The fix is idempotency: make "do this once" safe to call twice. Two practical patterns:

• Idempotency key. Generate one stable key per logical user request and attach it to any downstream side-effecting call. The receiving system dedupes on that key, so a retry that repeats the action is recognized and ignored. Reuse the same key across the original attempt and its retries.
• Commit on completion only. Don't perform side effects from inside the stream loop as tokens arrive. Buffer the model's output, wait for the terminal event, and only then execute the action. A stream that dies mid-way performed no side effect, so retrying it is clean by construction.

import time, uuid
import anthropic

client = anthropic.Anthropic()

def stream_with_retry(prompt: str, max_attempts: int = 3):
    request_key = str(uuid.uuid4())  # SAME key for all attempts of this request
    for attempt in range(max_attempts):
        text = ""
        try:
            with client.messages.stream(
                model="claude-opus-4-8",
                max_tokens=1024,
                messages=[{"role": "user", "content": prompt}],
            ) as stream:
                for chunk in stream.text_stream:
                    text += chunk
                final = stream.get_final_message()      # confirms the terminal event
            # Reached only on a clean finish. Do side effects HERE, keyed by
            # request_key so a prior partial attempt can't double-act.
            commit_result(text, idempotency_key=request_key)
            return text
        except anthropic.APIError:
            if attempt == max_attempts - 1:
                raise                                   # give up; surface to the user
            time.sleep(2 ** attempt)                     # 1s, 2s, 4s backoff
    return None

// Going deeper

The patterns above cover the common cases. A few advanced wrinkles matter once you run streaming at scale or in a relay architecture.

The proxy-buffering trap masquerades as a stall

Before you blame the model for an idle stall, rule out your own stack. A reverse proxy with response buffering on, a gzip layer, a serverless gateway, or a CDN can hold the token stream and release it in clumps — which looks exactly like a stall, then a burst. When debugging "my stream froze," test directly against the provider first to confirm the stream is healthy, then bisect each hop. A real mid-stream error and a buffering artifact need opposite fixes, so don't conflate them.

Relays multiply the failure surface

Most production apps don't stream the model straight to the browser — a server sits in the middle, consuming the model stream and re-emitting its own SSE feed to the client. That means there are now two streams that can die independently: model→server and server→browser. A clean strategy is to make the server the source of truth: it buffers what the model produced, and if the browser connection drops, the client reconnects and the server replays from its buffer rather than re-calling the model. This turns an expensive model retry into a cheap buffer replay for the browser-side failure.

Distinguish retryable from terminal failures

Not every mid-stream death deserves a retry. A transient network drop or a transient overload (529-style) is worth retrying. A malformed request, an auth failure, or a content refusal will fail identically every time — retrying just wastes money and time. Inspect the failure: if the provider sent an in-band error event, its type tells you whether it's retryable; if it was a raw transport drop, treat it as retryable but under the cap. When in doubt, fail loud and let a human or a higher layer decide, rather than looping.

Backpressure on slow consumers

A separate-but-related reliability concern: if your downstream consumer (a browser, a database write) can't keep up with how fast tokens arrive, chunks pile up in memory. Production relays respect backpressure — they pause reading from the model when the consumer's buffer is full — using the streaming primitives the platform provides rather than pushing every chunk into an unbounded array. An out-of-memory crash mid-stream is just another way to lose a response, and it's entirely self-inflicted.

The durable lesson: treat the terminal event, not the HTTP status, as your definition of success; salvage partials only when they're free-form text; and make every retry safe by keeping side effects out of the token loop. Get those three right and a dropped connection becomes a recoverable hiccup instead of a frozen, double-charged mess.