API Timeout: Find the Failing Layer and Retry Safely

An API timeout does not prove that the model is down. It means one participant stopped waiting: the client could not establish a connection, no data arrived before a read timeout, a total deadline expired, a proxy closed an idle stream, or a gateway did not receive an upstream response. Change only the layer supported by evidence.

Map the request path

Application / SDK
  → DNS and TCP/TLS
  → forward or reverse proxy
  → API gateway
  → upstream model
  → streamed response back to the client

Record start and end timestamps, the exception class, HTTP status and body, and a request identifier when the API returns one. Claude responses, for example, include a request-id; the official error guide recommends using it when investigating a specific request.

Distinguish four timeout types

Type What expired Signal Evidence-based change
Connect timeout DNS, TCP, or TLS setup No HTTP status; connection, SSL, or proxy error Check DNS, certificates, firewall, and proxy first
Read or idle timeout No bytes or stream events arrived Connection exists but the client or proxy closes it Check streaming, response size, and every idle timer
Total deadline Budget for the complete operation Client cancels regardless of phase Split the work or adjust only the justified deadline
Upstream timeout Gateway stopped waiting for its provider HTTP status and gateway/upstream error body Check provider status, queueing, and whether retry is allowed

There is no universal timeout value. A read timeout suitable for classification may be too short for a long streamed generation, while a very large deadline can hide a stuck queue and hold client resources.

For BetterToken requests, the API documentation and Dashboard provide an API-layer observation point: time, model, status, input/output/cache tokens, and cost. They do not replace client DNS/TLS logs and cannot prove where a third-party request stopped.

Diagnose from the client outward

First capture the SDK's connect timeout, read timeout, total deadline, automatic retries, and streaming mode. Do not rely on remembered defaults; SDK versions change.

OpenAI's error guide separates APITimeoutError from APIConnectionError, which covers network, proxy, SSL, and firewall problems. If no HTTP response exists, inspect those layers before blaming upstream generation.

Run the test from the same container or server as production. If a laptop succeeds but the workload fails, compare DNS, CA certificates, proxy variables, and egress rules. For a reverse proxy, inspect its own connect, read, and idle timers. Increasing the SDK timeout does not extend a shorter proxy timer.

If an HTTP status and error body exist, the request reached at least one server. Preserve the response and request ID. Do not merge timeout, 429, and context-overflow handlers without their distinct response signals.

For long Claude operations, Anthropic recommends streaming or Message Batches. Streaming can reduce some idle-timeout failures but does not guarantee completion or remove the application's total deadline.

Run two controlled tests

Keep the model, endpoint, key, and network unchanged.

  1. Short response test: send a minimal prompt with a short expected output. Record connection time, time to first byte, status, and total duration. If it fails, inspect connection, authentication, endpoint, and availability before increasing output.
  2. Controlled long response test: after the short test succeeds, change only output size or restore the original scenario. If the connection opens but the long response fails, compare streaming, read/idle timeout, total deadline, and proxy behavior.

Changing the model, network, timeout, and prompt simultaneously may hide the cause even if the next request succeeds.

Apply the smallest fix

  • Adjust connect timeout only when connection establishment is demonstrably slower.
  • Adjust read or idle timeout when a stream begins but an intermediary closes a data gap.
  • Increase the total deadline only when the business operation is expected to take longer and lower layers are healthy.
  • Reduce context or output when measured generation size is the cause.
  • Retry only errors documented as temporary, with bounded attempts and total time.

Before retrying a create, send, charge, or job-start request, ask whether the server could have completed it while the client lost the response. Check the destination state or use an endpoint-supported idempotency mechanism.

Verify the repair

The fix is complete only when the controlled tests confirm the expected result at the same endpoint and the original operation has not been duplicated.

Recovery checklist

  1. Preserve the timeout class, timestamps, status/body, and available request ID.
  2. Check the actual operation result before a retry.
  3. Bound attempts and the overall retry deadline.
  4. Account for retries performed by the SDK, proxy, and application.
  5. Re-run the short test.
  6. Run the original scenario once and compare status, duration, and result.

If the short request works but the controlled long request fails at the same point, do not keep expanding timeouts. Find the earliest layer that stops waiting and change that layer or the shape of the work.

Sources

Related articles