Retries
Which errors to retry, how long to wait and code that does it right.
What to retry
| Answer | Retry? | How |
|---|---|---|
429, 503, 529 | Yes | After Retry-After-Ms |
500, 502, 504 | Yes | After a short backoff, 1 to 2 seconds |
400, 401, 402, 403, 404, 413 | No | The same call fails the same way. Fix it first |
| Network error before any answer | Yes | After a short backoff |
| Error inside a stream | Usually | Resend the whole call. You paid only for what was written |
The gateway never retries a call itself: your SDK does, so tries don't multiply.
The official SDKs already do it
The OpenAI and Anthropic SDKs retry 429 and 5xx answers twice by default and wait for Retry-After-Ms. For long unattended jobs, raise it:
const client = new OpenAI({
baseURL: "https://binference.io/api/v1",
apiKey: process.env.BINF_API_KEY,
maxRetries: 5,
});Doing it yourself
With plain fetch, read the wait from the headers and add a little randomness:
async function call(body: unknown, tries = 5): Promise<Response> {
for (let attempt = 1; ; attempt++) {
const response = await fetch("https://binference.io/api/v1/chat/completions", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.BINF_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
});
const retryable = response.status === 429 || response.status >= 500;
if (!retryable || attempt >= tries) return response;
const named = Number(response.headers.get("retry-after-ms"));
const wait = Number.isFinite(named) && named > 0 ? named : 1_000 * 2 ** (attempt - 1);
await new Promise((resolve) => setTimeout(resolve, wait + Math.random() * 250));
}
}Time limits
- Streamed calls: up to 30 minutes.
- Calls that aren't streamed: up to 13 minutes.
- At the limit, the call ends with a clean error and is charged for what was written.
Set your client's timeout above these, or stream: some SDKs give up on a silent connection after 10 minutes.