HTTP 429 Too Many Requests: Safe Retry-After Parsing and Exponential Backoff

Understanding HTTP 429 in LLM APIs

When integrating large language model APIs (such as OpenAI, Anthropic, or compatible multi-model gateways), 429 status codes typically stem from three distinct mechanisms:

1. RPM (Requests Per Minute): Triggered when concurrent workers fire too many parallel requests without queue orchestration.
2. TPM (Tokens Per Minute): Triggered when combined prompt context and generation tokens exceed rolling minute-by-minute token budgets.
3. Quota or Balance Exhaustion: Hard limits resulting from exhausted prepaid credits, zero balance, or strict monthly spend caps.

HTTP/1.1 429 Too Many Requests
Date: Sun, 23 Aug 2026 03:00:00 GMT
Content-Type: application/json
Retry-After: 6
x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 6s
x-ratelimit-limit-tokens: 30000
x-ratelimit-remaining-tokens: 1200
x-ratelimit-reset-tokens: 150ms

{
  "error": {
    "message": "Rate limit reached for model in organization on tokens per minute (TPM). Please try again in 6s.",
    "type": "tokens",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

When an error is caused by credit exhaustion or rigid 5-hour subscription locks, automated retries waste network bandwidth and worker threads. To immediately isolate the root cause instead of parsing cryptic error payloads, BetterToken provides an observability Dashboard: it displays real-time HTTP status codes, granular breakdown of input, output, and cache tokens per request, and transparent pay-as-you-go balances without disruptive 5-hour rolling lockout windows.

HTTP 429 Diagnostic Matrix

Correctly Parsing the Retry-After Header

According to RFC 6585, the Retry-After response header communicates the required wait interval in two standard formats:

• Relative seconds (integer or decimal, e.g., Retry-After: 12);
• HTTP-date timestamp (e.g., Retry-After: Sun, 23 Aug 2026 03:05:00 GMT).

import datetime
import email.utils
import time

def parse_retry_after(header_value: str | None, default_delay: float = 1.0) -> float:
    if not header_value:
        return default_delay
    header_value = header_value.strip()
    try:
        # Parse numeric seconds
        return max(0.0, float(header_value))
    except ValueError:
        pass

    try:
        # Parse RFC HTTP-date
        parsed_date = email.utils.parsedate_to_datetime(header_value)
        now = datetime.datetime.now(datetime.timezone.utc)
        delay = (parsed_date - now).total_seconds()
        return max(0.0, delay)
    except Exception:
        return default_delay
Show more

Implementing Full Jitter Exponential Backoff

When the Retry-After header is absent, use exponential backoff enhanced with Full Jitter. The wait duration for attempt $i$ is calculated as:

$T_{ ext{wait}} = ext{random}(0, \min(T_{ ext{max}}, T_{ ext{base}} imes 2^i))$

Randomizing delay spreads out retries across workers, breaking concurrent lockstep and preventing server overload.

import asyncio
import json
import random
import httpx

class RateLimitRetryClient:
    def __init__(
        self,
        base_url: str = "https://www.bettertoken.ai/v1",
        api_key: str = "",
        max_retries: int = 4,
        base_delay: float = 1.0,
        max_delay: float = 32.0,
    ):
        self.base_url = base_url
        self.api_key = api_key
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.client = httpx.AsyncClient(
            base_url=self.base_url,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=60.0,
        )

    async def send_chat_completion(self, payload: dict) -> dict:
        for attempt in range(self.max_retries + 1):
            try:
                response = await self.client.post("/chat/completions", json=payload)
                
                if response.status_code == 200:
                    return response.json()

                if response.status_code == 429:
                    error_data = response.json().get("error", {})
                    error_code = error_data.get("code")

                    # If balance/quota is depleted, retrying is futile
                    if error_code in ("insufficient_quota", "billing_not_active"):
                        raise RuntimeError(f"Billing exhaustion: {error_data.get('message')}")

                    if attempt == self.max_retries:
                        raise RuntimeError(f"Max retries exhausted (429): {response.text}")

                    # Determine wait time
                    retry_after = response.headers.get("Retry-After")
                    if retry_after:
                        wait_time = parse_retry_after(retry_after) + random.uniform(0.1, 0.5)
                    else:
                        # Full Jitter Exponential Backoff
                        backoff_cap = min(self.max_delay, self.base_delay * (2 ** attempt))
                        wait_time = random.uniform(0, backoff_cap)

                    await asyncio.sleep(wait_time)
                    continue

                response.raise_for_status()

            except httpx.RequestError as exc:
                if attempt == self.max_retries:
                    raise
                wait_time = min(self.max_delay, self.base_delay * (2 ** attempt))
                await asyncio.sleep(wait_time)

        raise RuntimeError("Request failed after exhausting retry budget")
Show more

Idempotency and Side-Effect Safety

Retrying read-only operations (GET) is inherently idempotent. However, when invoking LLM inference or agent tasks via POST:

1. Prevent Duplicate Agent Execution: If a network timeout occurs mid-generation, inspect whether tokens were consumed or output persisted before blindly re-dispatching.
2. Assign Client Request IDs: Include unique X-Request-ID headers to trace retried calls in upstream logs.
3. Never Treat 401/403 as 429: Authentication errors cannot be resolved by backoff loops; they require credential renewal.

Verifying Recovery

Before resuming high-throughput workloads:

• Dispatch a minimal health-check probe (max_tokens: 5).
• Verify HTTP 200 and inspect x-ratelimit-remaining-requests.
• Gradually ramp up concurrency while tracking 429 error ratios in monitoring dashboards.

To eliminate sudden 429 bottlenecks caused by rigid minute limits and maintain complete visibility into request status, switch to BetterToken API, generate dedicated API keys, and monitor real-time token telemetry in the Dashboard.

Primary References

• RFC 6585: Additional HTTP Status Codes (429 Too Many Requests)
• OpenAI Guide on Rate Limits & Error Handling
• BetterToken API Reference & Documentation