Rate limits, retries and the backoff you actually need
Rate limits are a normal operating condition for AI features. Exponential backoff with jitter, respecting the retry-after header, distinguishing retryable from permanent errors, and a circuit breaker for sustained failure turn a recurring incident into an invisible one.
Rate limits are not an edge case. They are a scheduled part of operating an AI feature, and the difference between a hiccup and an incident is entirely in code you write before the first one.
Backoff, with jitter
Retry after one second, then two, then four, then eight. Exponential backoff gives the upstream room to recover instead of adding to the problem.
Then randomise each delay by some percentage. Without jitter, every client that hit the limit at the same moment retries at the same moment — which is the thundering herd that caused the limit in the first place.
Prefer the provider's number
When a response carries a retry-after header, use it. It reflects something the provider knows about its own capacity that your backoff curve is guessing at.
Not everything should be retried
A 429 or a 503 is temporary and worth retrying. A 400 is a malformed request, and it will be exactly as malformed on the fourth attempt. A 401 is a credential problem that retrying will not solve.
Classify errors explicitly. Retrying everything wastes time and money and turns a clear error into a slow, confusing one.
Fail fast when it is really down
If the provider has been failing for two minutes, the next request will fail too. A circuit breaker that trips after a run of failures and rejects immediately for a cool-down period stops you queueing work that cannot succeed.
It also protects the rest of your system: without it, an upstream outage becomes a backlog of threads waiting on timeouts.
Decide what the user sees
Background work can retry patiently and at length. Interactive work cannot — a person watching a spinner needs an answer or an honest failure within a few seconds.
Two different policies for two different contexts, chosen deliberately. The failure path is part of the feature, and it is the part that gets designed last if it gets designed at all.
Frequently asked questions
Why does jitter matter?
Without it every client that hit the limit retries at the same instant, recreating the spike that caused the limit. Randomising the delay spreads the load and is a two-line change.
How many retries is right?
Three to five for a background job. For anything a user is waiting on, one or two — then fail with something honest. A user staring at a spinner for forty seconds is worse served than one told to try again.
