How to fix: request timeout on large or streaming responses

How it shows up

The same root cause surfaces in a few different ways. If any of these match, you're in the right place.

• ▸Short prompts succeed and long ones fail — the boundary tracks output length, not input.
• ▸The request fails at a suspiciously round number of seconds (30, 60, 120), which points at a fixed timeout rather than the model.
• ▸The failure is a client-side or proxy timeout, not an error from the provider — you never receive a response body.
• ▸It works locally but times out behind a load balancer, CDN, or API gateway.
• ▸A streaming request dies mid-response after a period of silence, typically while the model is reasoning before emitting tokens.

Diagnose it first

Confirm which cause you actually have before changing anything — most wasted time here comes from fixing the wrong one.

Provider-specific behaviour

The error means roughly the same thing everywhere, but each provider signals it differently.

Reasoning models

Models that reason before answering can be silent for a long stretch before the first token appears. An idle-timeout policy measuring time-between-bytes will kill these connections even though the request is healthy — the connection is idle by design during reasoning.

Proxies and load balancers

Default idle timeouts on common reverse proxies and cloud load balancers are frequently around 60 seconds, well below a long generation. Raising the SDK timeout alone changes nothing, because the connection is severed upstream of your code.

Serverless platforms

Serverless functions have a hard maximum execution duration that you cannot exceed regardless of client configuration. Long generations need either streaming with an appropriate response mode, or a background job that the client polls.

Batch endpoints

Where latency is not user-facing, provider batch APIs sidestep request timeouts entirely by design — work is submitted and collected asynchronously rather than held open on a connection.

What doesn't work

Common fixes for this error that look right and aren't.

• Raising the SDK timeout and nothing else.

  If a proxy or load balancer between you and the provider has a lower limit, it terminates the connection first. The SDK setting only binds when it is the smallest timeout in the chain.

• Retrying the timed-out request unchanged.

  The same request takes the same time and hits the same limit, so it fails identically — while you pay for the tokens generated on every attempt before it was cut off.

• Lowering max_tokens to force the response under the limit.

  This does prevent the timeout, but by truncating output mid-answer. You have traded a visible error for silent, hard-to-detect quality loss.

Common variations and related errors

You'll usually hit this same root cause under a few different names. Same fix.

• ▸"request timeout"
• ▸"ETIMEDOUT"
• ▸"upstream timeout"
• ▸"gateway timeout (504)"
• ▸"context deadline exceeded"
• ▸fetch failed: timeout

Code: drop-in retry helper

A minimal typescript helper that handles the most common version of this error. Copy, paste, adjust the policy to your traffic.

// Stream long generations instead of waiting for the full response.
import OpenAI from 'openai';
const client = new OpenAI();

const stream = await client.responses.create({
  model: 'gpt-4o',
  input: prompt,
  max_output_tokens: 16000,
  stream: true,
});
for await (const event of stream) {
  if (event.type === 'response.output_text.delta') {
    process.stdout.write(event.delta);
  }
}