API Timeouts: Find the Breakpoint and Retry Safely
Separate connect, read, deadline, and upstream timeouts, test each layer, and retry without duplicating a completed operation.
An API timeout does not prove that the “model is down.” It means one participant in the chain stopped waiting: the client failed to connect, waited too long for data, exhausted the overall deadline, a proxy closed an idle connection, or the gateway did not receive a response from upstream. Fix only the layer you have confirmed instead of increasing every timeout at once.
Where a request can break
The same user-facing “request timed out” message can originate at different points. Record the start and end times, exception class, HTTP status, and response body. If the API returns a request identifier, keep that too. In the Claude API, for example, every response includes request-id; the official documentation recommends using it when investigating a specific request.
Four different timeouts
There are no universal values. The same read timeout can be reasonable for a short classification and too low for a long streaming response. Conversely, an enormous overall deadline can hide a stuck queue while holding client resources.
For requests routed through BetterToken, the API documentation and Dashboard let you correlate the time and status of your request with the model, input/output/cache tokens, and charge. This is a useful API-layer observation point, but it does not replace client-side DNS/TLS logs or prove where a third-party service's request broke.
Diagnose each layer
1. Client and SDK
First, capture the actual settings: connect timeout, read timeout, overall deadline, automatic retries, and streaming mode. Do not rely on your memory of “library defaults”—an SDK version may have changed them.
OpenAI describes APITimeoutError as a request timeout and classifies network, proxy, SSL, and firewall problems as APIConnectionError. That distinction is useful: receiving no HTTP response calls for different checks than an error with a code and body.
2. DNS, TLS, and proxy
Send the request from the same environment where the application runs. If it succeeds locally but fails in a container or on a server, compare DNS, CA certificates, proxy variables, and the outbound firewall. Check the reverse proxy's connect/read/idle timers separately: increasing the SDK timeout will not extend an intermediary's wait.
3. Gateway and upstream
If you received an HTTP status, this is no longer a pure connect timeout. Save the error body and request identifier. Do not lump a timeout together with a 429 or context overflow; they have their own response signals.
For long Claude requests, Anthropic recommends streaming or Message Batches, especially when processing may take a long time. Streaming can prevent some idle timeouts, but it does not guarantee completion or override the application's overall deadline.
Two tests that narrow down the cause
Run the tests in sequence while keeping the model, endpoint, network, and key unchanged.
Test A: short, controlled response
Send a minimal request with a short expected output. Record the connection time, time to first byte, complete HTTP status, and duration. If this test fails, increasing the response size is pointless: check the network, authentication, endpoint, and API availability.
Test B: controlled long response
After Test A succeeds, increase only the expected output size or enable the original workload. If the connection is established but the long response breaks, compare streaming, read/idle timeout, overall deadline, and proxy behavior. Do not change the model, network, and timeouts together, or the cause will remain unknown.
When to change a setting
- Change the connect timeout only if DNS/TCP/TLS genuinely takes longer to establish, not when upstream generation is slow.
- Change the read or idle timeout when the stream starts but an intermediary closes the connection during a gap between data events.
- Increase the overall deadline when the business operation is legitimately longer and all lower layers are working. For an oversized task, consider splitting it up or using an asynchronous mode first.
- Reduce
max_tokensand context size when the delay comes from a long generation, but do not use this as an explanation without measurements. - Retry only errors that the documentation identifies as transient, and cap both the number of attempts and the total time budget.
Safe retries without duplicate operations
Before resending, answer this question: could the server have completed the request while the client lost the response? The risk is usually lower for a status read. Before retrying record creation, task execution, message delivery, or a charge, check the actual result or use the endpoint's supported idempotency mechanism.
Minimal checklist:
- Save the timeout class, timestamps, status/body, and any request ID.
- Check the operation's result in the target system before retrying.
- Limit the number of attempts and the overall retry deadline.
- Do not stack retries in the SDK, proxy, and application without accounting for the total number of HTTP requests.
- Run the short Test A after the fix.
- Then run the original Test B once and compare status, duration, and result.
If the short request succeeds but the controlled long request breaks in the same place again, that is not a reason to increase the timeout without limit. Find the earliest layer that stops waiting, then change only that layer or the form of the work.
Short conclusion
Diagnose an API timeout from observable signals: no HTTP response means checking the connection; a status and body mean checking the gateway or upstream; a long stream that breaks means comparing streaming and idle timeouts; an exhausted total budget means changing the architecture or overall deadline. Confirm recovery with two tests, and do not repeat a risky operation until you have checked its actual outcome.