API Timeout: How to Diagnose Dropouts and Configure Safe Retries for Long LLM Generations

Where Drops Occur: HTTP Request Lifecycle Phases

An API request to a large language model consists of four distinct operational phases, each requiring independent timeout configuration:

[Client] --- (1. Connect Timeout) ---> [API Gateway]
[Client] --- (2. Write Timeout)   ---> [Prompt Upload]
[Model]  --- (3. Read / TTFT)     ---> [Reasoning & Response]
[Client] <--- (4. Pool Timeout)    --- [Connection Pool Holding]

1. Connect Timeout: Time allocated to establish TCP connections and complete TLS handshakes (recommended 5–10 seconds).
2. Write Timeout: Time required to transmit the request payload (critical when sending 100k+ token context windows).
3. Read Timeout: Window spent waiting for the server response or successive chunks in an SSE stream.
4. Pool Timeout: Duration spent waiting for an available socket from the client connection pool under high concurrency.

---

Timeout Diagnostics Matrix

---

Granular Timeout Configuration in Python (HTTPX)

Defaulting to timeout=10.0 in standard client libraries inevitably causes drops when calling reasoning models. Here is a resilient client configuration:

import os
import httpx
from openai import OpenAI

API_KEY = os.environ.get("BETTERTOKEN_API_KEY", "your_api_key_here")

custom_timeout = httpx.Timeout(
    connect=5.0,    # Fast failure on unreachable network
    read=120.0,     # Ample window for extended reasoning
    write=10.0,     # Time to transmit prompt payload
    pool=10.0       # Pool socket acquisition window
)

http_client = httpx.Client(
    timeout=custom_timeout,
    limits=httpx.Limits(max_keepalive_connections=50, max_connections=100)
)

client = OpenAI(
    base_url="https://www.bettertoken.ai/v1",
    api_key=API_KEY,
    http_client=http_client
)

response = client.chat.completions.create(
    model="claude-3-7-sonnet-20250219",
    messages=[
        {"role": "system", "content": "You are a senior systems architect."},
        {"role": "user", "content": "Design a high-throughput distributed message broker."}
    ],
    stream=True
)

for chunk in response:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
Show more

---

Stabilizing SSE Streams and Safe Retries

To avoid duplicate costs and API hammering during failures, retries must adhere to three foundational rules:

1. Never retry if streaming has already started: If partial tokens have been received, re-sending the whole prompt will lead to double billing.
2. Exponential Backoff with Full Jitter: Retries must be spaced using progressive, randomized delays to prevent thundering herd problems.
3. Idempotency Keys: Use unique task IDs in background batch jobs to prevent duplicate generations.

import time
import random
import httpx

def execute_with_safe_retry(client, model, messages, max_retries=3):
    base_delay = 1.0
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                stream=True
            )
            return response
        except (httpx.ConnectTimeout, httpx.ReadTimeout, httpx.NetworkError) as err:
            if attempt == max_retries - 1:
                raise err
            sleep_time = random.uniform(0, base_delay * (2 ** attempt))
            time.sleep(sleep_time)

Verification Checklist

• Response status returns 200 OK.
• SSE stream receives all generation chunks without ChunkedEncodingError.
• Sockets are cleanly returned to the pool after stream completion.

For further architecture guidelines and gateway parameters, visit the BetterToken API Reference.