Batch inference (Batch API)
Last verified: June 2026· definition
Most major providers offer a batch endpoint at roughly half the standard per-token price, with results returned within a window (commonly up to 24 hours) rather than immediately. You trade latency you do not need for money you would rather keep.
The fit is anything not sitting in front of a waiting user: nightly enrichment, backfills, generating embeddings over a corpus, running an eval suite, bulk classification, offline summarization. In most production systems a surprising share of total token volume is exactly this kind of work, still being paid for at interactive prices.
The engineering cost is real but modest: you need a job queue, result reconciliation keyed by request ID, and a retry path. It is usually the least controversial cost lever available, because nothing user-facing changes.
How much of your traffic is secretly batchable
The question worth asking is not whether you have batch workloads but which of your current interactive calls have no user waiting on them. Nightly enrichment, embedding generation over a corpus, eval suite runs, bulk classification, backfills, and scheduled summarisation are all commonly implemented against the standard synchronous endpoint because that is what the code already did. Auditing routes by whether a human is actually blocked on the response usually surfaces more batchable volume than teams expect.
The engineering shape is different, not harder
Batch work is submitted as a job and collected later, which means the calling code cannot simply await a response. You need somewhere to persist the job identifier, a way to poll or be notified on completion, and handling for partial failure — individual requests within a batch can fail while the job succeeds. None of this is difficult, but it is genuinely different plumbing, and it is the reason batch adoption tends to lag its economics.
Where batch is the wrong answer
Anything with a user waiting, obviously. Less obviously: work whose inputs change faster than the completion window, since results can arrive already stale. Also anything on a critical path with a tight deadline, because completion windows are upper bounds rather than guarantees and you cannot rely on finishing early. If a delay of hours would be a problem, the discount is not available to you.
Common misconceptions
MythBatch inference uses a weaker model.
RealityIt is the same models. What differs is the delivery guarantee — asynchronous completion within a window instead of an immediate response. Output quality is not part of the trade.
MythBatch is only worth it at very large volume.
RealityThe discount is proportional, so it applies at any volume. What varies is whether the integration effort is worth the saving. For a recurring job the work is done once and the saving recurs indefinitely.
MythYou have to move an entire workload to batch.
RealityRouting is per-call. The common pattern is a single application serving interactive traffic synchronously while its background jobs go to the batch endpoint, with the same prompts and the same models.
Frequently asked questions
How much does batch inference actually save?
Major providers price batch endpoints at a substantial discount to standard synchronous rates — commonly around half, though the exact figure varies by provider and model, so check current pricing rather than relying on a remembered number. Because the discount is proportional, the saving on any given workload is straightforward to calculate: whatever that work costs today, multiplied by the discount.
What happens if a batch job fails partway through?
Providers generally return per-request results, so individual failures surface as failed entries within an otherwise successful job rather than failing the whole batch. Your collection code needs to handle a partial result set — reconciling what succeeded, retrying what did not. Treating the job as all-or-nothing is the most common integration bug.
Can I use batch for embedding a large corpus?
Yes, and it is one of the clearest fits. Corpus embedding is high volume, entirely offline, and has no latency requirement, which is exactly the profile the pricing rewards. It is often the single largest batchable line item in a retrieval system, and frequently the first place teams look when the initial indexing bill arrives.