![\"A](\"/10186e254c7300e74cb1f561b240de8e/retry-flow.svg\")
The first time a rate limit bites, it never looks like a rate limit. A demo that ran fine all week suddenly throws 429 in the middle of a live session. An agent that was happily calling tools stalls halfway through a task. A batch of documents you are indexing gets through the first forty, and then every call comes back rejected. Nothing in your code changed. What changed is that more requests arrived at once than the provider is willing to serve you in a minute.
This post is for engineers running an LLM-backed service, especially an async FastAPI one, who have hit that wall or want to stop hitting it. I’ve run into it in the backends behind CloudCanvasAI, Archi, and Gemini Alchemy. In all three, a single user request fans out into many model calls, and a burst of those calls trips the provider’s per-minute budget.
\nThe fix is not one thing. It is a retry policy that knows what to retry, a backoff that does not make the problem worse, and a way to pace your own traffic so you rarely reach the limit at all. The sections below take those in order.
\n\n
Most people picture a rate limit as “requests per minute” and stop there. LLM providers enforce at least two limits at once, and the second one is the one that surprises you.
\nThe token limit is why counting requests alone misleads you. A request counter sees a small chat turn and a call that stuffs a 100k-token document into a long-context model as the same thing: one request each. But the second one can blow your token budget on its own. If you pace only by request count, you will still get 429s from the token side, and you will wonder why, because your dashboard shows you nowhere near the request ceiling.
Both limits report back the same way: an HTTP 429 with headers telling you where you stand. The response carries retry-after (how many seconds to wait) and a set of x-ratelimit-* headers showing your limit and how much of it is left. Those headers are not decoration. They are the provider telling you exactly how long to back off, and the single most common mistake is ignoring them.
Before writing any backoff code, sort the failures. Retrying blindly is how a slow minute becomes an outage.
\n
The status code decides which side of the line a failure falls on:
\n400, 401, 403, 404: do not retry. These are your fault, not a timing problem: a malformed request, a bad key, a missing permission, or a wrong model ID. A retry sends the identical broken request and gets the identical rejection. Fail fast and surface it.429, 500, 502, 503, 504, 529, plus timeouts and dropped connections: retry. The 429 is a rate limit. 500 and 502/503/504 are transient server faults. 529 is Anthropic’s “overloaded” signal, which means the service is busy rather than your account being over budget. All of these can succeed if you wait and try again.The Anthropic error reference marks which codes are retryable, and the mapping is close enough across providers that you can write one classifier. In the Anthropic Python SDK, the exception types make this clean. anthropic.RateLimitError (429), anthropic.InternalServerError (5xx), and anthropic.APIConnectionError (the network dropped before a response) are the retryable branch. An anthropic.BadRequestError or anthropic.NotFoundError is not.
When a call is retryable and there is no retry-after to obey, you wait before trying again, and you wait longer each time. That is exponential backoff: one second, then two, then four. Most people know that part. The part they skip is jitter (randomness in the delay), and skipping it turns a recoverable blip into a self-inflicted stampede.
Think about what caused the 429 in the first place: a burst of requests all landed together. If every one of them backs off by exactly the same formula, they all wake up at the same instant and hit the API together again. You have rebuilt the original burst, just shifted a few seconds later. AWS wrote the reference piece on this, Exponential Backoff And Jitter, and its conclusion is blunt: add randomness to the delay so the retries spread out instead of synchronizing.
The version I reach for is full jitter. Instead of sleeping the full exponential delay, sleep a random amount between zero and that delay:
\nimport random\n\nBASE = 1.0 # seconds\nCAP = 60.0 # never wait longer than this between tries\n\ndef backoff_seconds(attempt: int, retry_after: float | None = None) -> float:\n # The server told us how long to wait. Obey it exactly.\n if retry_after is not None:\n return retry_after\n # Otherwise: full jitter over an exponentially growing window.\n window = min(CAP, BASE * 2 ** attempt)\n return random.uniform(0, window)Two rules do the work here:
\nretry-after is present, it wins outright. The provider has told you the exact wait. Guessing a longer exponential delay wastes time, and guessing a shorter one earns another 429. The Retry-After header exists precisely so you do not have to guess.Next, wrap that function around a call. The loop below adds error classification and a hard attempt ceiling, and it pulls retry-after from the error’s response when there is one:
import asyncio\nimport anthropic\n\nRETRYABLE = (\n anthropic.RateLimitError, # 429\n anthropic.InternalServerError, # 500, 502, 503, 504, 529\n anthropic.APIConnectionError, # network dropped before a response\n)\n\nasync def call_with_retry(make_call, *, max_attempts: int = 6):\n for attempt in range(max_attempts):\n try:\n return await make_call()\n except RETRYABLE as err:\n if attempt == max_attempts - 1:\n raise # out of tries; let the caller decide what to do\n retry_after = None\n resp = getattr(err, \"response\", None)\n if resp is not None:\n header = resp.headers.get(\"retry-after\")\n retry_after = float(header) if header else None\n await asyncio.sleep(backoff_seconds(attempt, retry_after))The ceiling matters as much as the backoff. A retry loop with no limit does not heal an overloaded endpoint. It keeps throwing load at the one service that is already past its budget. Meanwhile, every request waiting on that loop holds a connection, a task, and a slice of your event loop. Bound the attempts, and bound the total wall-clock time too if the call sits on a request path where a user is waiting.
\nYou rarely need to write that loop by hand, because the official SDKs already do it. The Anthropic SDK retries 408, 409, 429, and 5xx responses plus connection errors, with exponential backoff, and it reads retry-after for you. The only catch is the default: two retries. The snippet below raises that budget for a whole client, or for a single call:
from anthropic import Anthropic\n\n# Raise the retry budget for the whole client.\nclient = Anthropic(max_retries=5)\n\n# Or override per request, e.g. on a call you know is bursty.\nclient.with_options(max_retries=8).messages.create(\n model=\"claude-opus-4-8\",\n max_tokens=1024,\n messages=[{\"role\": \"user\", \"content\": \"...\"}],\n)Two retries is fine for a quiet service and thin under real fan-out, so set it deliberately rather than inheriting the default. Watch the timeout as well. The SDK’s client timeout defaults to ten minutes, and timeouts are themselves retried. So the worst-case wall-clock time for a call is roughly the timeout times the retry count, which is worth knowing before you set max_retries to something large on a user-facing path.
The same knob exists elsewhere. If you call the API over raw httpx instead of the SDK, tenacity gives you the same retry-with-jittered-backoff behavior as a decorator. OpenAI’s SDK exposes the identical max_retries setting.
Reaching for the SDK’s retry is the right default. Write your own loop only when you need behavior it does not give you: a shared budget across many concurrent calls, a circuit breaker, or logic that switches to a fallback model after N failures instead of retrying the same one.
\nRetries are damage control. The better lever is pacing your own traffic so you stay under the limit on purpose. That matters most exactly where I hit rate limits: fan-out, where one job launches many model calls. Think of indexing a document set, enriching every chunk with a model-generated summary, or running an agent that issues many tool calls. The naive version launches all of them at once and lets the provider sort it out. The provider sorts it out with a wave of 429s.
![\"Two](\"/66924860f5c6bd2771922fd08b7ea3ce/concurrency-gate.svg\")
A bounded concurrency gate is the smallest thing that helps. A semaphore caps how many requests are in flight at once, so the burst becomes a steady stream. In the code below, each chunk waits for a free slot in the gate, then makes its call through call_with_retry:
import asyncio\n\n# Tune this to your requests-per-minute budget, not to your CPU count.\ngate = asyncio.Semaphore(8)\n\nasync def summarize_chunk(client, text: str) -> str:\n async with gate: # at most 8 model calls in flight at any moment\n resp = await call_with_retry(lambda: client.messages.create(\n model=\"claude-opus-4-8\",\n max_tokens=256,\n messages=[{\"role\": \"user\", \"content\": f\"Summarize:\\n{text}\"}],\n ))\n return resp.content[0].text\n\nasync def summarize_all(client, chunks: list[str]) -> list[str]:\n return await asyncio.gather(*(summarize_chunk(client, c) for c in chunks))Pick the semaphore size from your request budget, not from how many tasks you happen to have. And remember the token side. Eight concurrent calls sounds safe until each one carries a large prompt, and then you are under the request limit and over the token limit.
\nFor that, the honest fix is a per-minute token allowance on top of the concurrency cap. A small async rate limiter such as aiolimiter gives you the request-per-window bucket, and you track token spend yourself against the TPM figure from the rate-limit headers. Staying comfortably under the ceiling costs you a little throughput. It buys you a service that does not spend half its time retrying.
Ignoring retry-after. The provider hands you the exact wait, and your code guesses a different number. Guess too short and you earn another 429 immediately; guess too long and you sit idle. Read the header first, always.
No jitter. Synchronized retries reproduce the original burst. Jitter is the single change that separates a retry policy that recovers from one that oscillates.
\nRetrying the non-retryable. A 400 for a malformed request or a 401 for a bad key will fail identically on every attempt. All you have added is latency and, on a paid endpoint, cost. Classify before you loop.
Expecting a retry to save a broken stream. Once you are streaming a response over SSE, an automatic retry cannot silently paper over a failure mid-stream, and the tokens already generated were already billed. Rate limits are checked when the request starts. So the durable answer for streaming under load is to keep concurrency low enough that streams start cleanly, and to fall back to another model on persistent 529 overload rather than hammering the busy one.
Paying for retries. Every accepted request bills its tokens. A retry that succeeds on the third try also billed the two failed attempts, if they got far enough to generate output. On a tight budget, aggressive retries quietly inflate spend. For work that is not latency-sensitive, the Message Batches API sidesteps the whole problem: you submit the jobs, the provider schedules them under its own limits at half the price, and you stop managing 429s by hand.
Early on, I treated rate limits as something to survive at the edge: wrap calls in a retry decorator and move on. That handles the symptom. What actually stabilized the fan-out paths in CloudCanvasAI was moving the control one level up, to a bounded gate in front of the model calls. Now the service shapes its own traffic instead of discovering the limit by bouncing off it. Retries still sit underneath as the safety net, but they fire rarely, because the gate keeps me under the ceiling to begin with.
\nThe other habit worth forming early is watching the x-ratelimit-* headers as a live signal, rather than waiting for the 429. They tell you how much budget is left this minute. If you log the remaining-tokens header and see it trending toward zero, you can slow down before the wall instead of after it. That turns rate limiting from an exception you catch into a number you steer by. It is the same instinct behind keeping the event loop unblocked: the smooth-looking service is the one that never lets a queue build up in the first place.
Rate limits are not an edge case to catch and forget. They are a budget the provider hands you every minute, in two dimensions, and a well-behaved client spends that budget on purpose. Obey retry-after when it is there. Add jitter so your retries spread out instead of firing in unison. Retry only what a second attempt can fix, cap the attempts, and put a bounded gate in front of your fan-out so the retry path stays the exception rather than the norm.
I have shipped every one of these behind the model calls in CloudCanvasAI, Gemini Alchemy, and the RAG copilot for CMS operations at CERN. The pattern that held up is the dull one: stay under the limit deliberately, and keep retries as the net you hope not to need. The services that survive a traffic spike are not the ones with the cleverest retry loop. They are the ones that saw the wall coming and slowed down first.
\nDiagrams by M. Hassan Ahmed, released under CC0. Image credit: original work by the author.
","frontmatter":{"title":"Handling LLM API Rate Limits: Retries and Backoff","date":"2026-07-11T00:00:00.000Z","description":"Your LLM backend returns 429s the moment traffic bursts. How to retry with backoff and jitter, respect Retry-After, and pace fan-out to stay under the limit.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABvElEQVQoz22Q2XLbMAxF9WQt3EktpERSu+TYdVwvrSeebE7+/58K2X3MzCHmcoBLEAieJrsBZjf15nrZfn+er5dfry+7z7cD8Pby/NC3tyPE9+v+fJjWYwkWMAYrWoe0AUD48fJ0+Oq3r8D6921z/F7vb8P2fXP4Gncf0+5j3t9Md1qRGgBLgKUj0i+oGgkfMou4S4SLuY2ZTRbtE9Dc4qLDqlFZy/Oepg2WPoDSmJYhNhGtuPA266SsU9VI1UAKMUuEp3ekGXnaclmzfIBXEmYDOFW7Gzd/lZnytJ7tAA2pcEL6wY5zs9V5K6VTyj/ignTatFR5MFci7zI7c9WKesr3zwgbqhyipflzEkWLhQUQt0hYIi1llenmbJgRKYOEljExEdYAjIqzJqEVfAexCmVNDHHhfr0LqE+Ux2mdgDmmJsR6gWjJqiZzLNVcwQpKSP1IRHSINLQMoKFvBuuHiJS5bsZ+TEhW6pzxggqNmVng5r+4a55a51umqiAmBVOOZzWF2UxbHLf9rtuchvV5KnsfwyzMLNBH1FhaYbq0GmAv0DlfJRkQoixCeZjkLK2KyqfaRbiA7A9AGVrEPzB4P5h5opdNAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/30ff3f68b1dcfb1160e7300152411747/40a76/hero.png","srcSet":"/static/30ff3f68b1dcfb1160e7300152411747/c972b/hero.png 340w,\n/static/30ff3f68b1dcfb1160e7300152411747/27625/hero.png 680w,\n/static/30ff3f68b1dcfb1160e7300152411747/40a76/hero.png 1360w,\n/static/30ff3f68b1dcfb1160e7300152411747/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-11-llm-api-rate-limits-retries-backoff/","previous":"blog/2026-07-12-llm-kv-cache-gpu-memory/","next":"blog/2026-07-14-reliable-json-from-llms/"}},"staticQueryHashes":["32046230"]}