Quantization
Last verified: June 2026· definition
A model’s weights are normally 16-bit floats. Quantizing to 8-bit or 4-bit reduces the memory footprint roughly proportionally, which means a model that needed multiple GPUs might fit on one, and throughput per GPU goes up. For self-hosted inference, that is the difference between viable unit economics and not.
Quality degrades gracefully rather than falling off a cliff: 8-bit is usually indistinguishable on most tasks, 4-bit is often fine for straightforward work and noticeably weaker on hard reasoning. The honest answer for any specific workload comes from an eval, not a blog post.
Quantization only matters if you run the weights yourself. If you call a hosted API, the provider has already made these choices for you — your cost levers are routing, caching, and token discipline instead.
What is actually being traded
Weights stored at lower numeric precision take less memory and are faster to move, which is what makes a model fit on smaller hardware and run more cheaply. The cost is a small loss of fidelity in each weight, which shows up as a quality change that is usually slight and is not uniform across tasks. Because the effect is task-dependent, the only way to know whether a given quantisation level is acceptable is to measure it on the work you actually do.
Memory is usually the binding constraint
For self-hosted inference, whether a model fits in available memory is often more decisive than raw speed, because a model that does not fit either cannot run or must be split across devices at a significant performance cost. Quantisation is frequently the difference between needing one accelerator and needing several, which changes the economics of a deployment more than the throughput improvement alone does.
Degradation is uneven
Quality loss tends to concentrate rather than spread evenly. Simple classification and extraction often survive aggressive quantisation with no measurable change, while multi-step reasoning, long-context work, and tasks requiring precise recall degrade sooner. An evaluation covering only easy cases will report that quantisation is free when it is not, so the eval set needs to include the hardest work the model is expected to do.
It interacts with the hardware decision
Because quantisation changes the memory footprint, it should be settled before hardware is specified rather than after. Choosing a quantisation level that meets your quality bar, then sizing hardware for that, avoids the common sequence where capacity is bought for full precision and then the model is quantised anyway, leaving expensive capacity underused.
Common misconceptions
MythQuantisation is free quality-wise.
RealityThe loss is often small enough not to matter, which is not the same as absent. Whether it matters depends on the task, and only evaluation on your workload settles it.
MythMore aggressive quantisation is always worth the saving.
RealityDegradation accelerates at lower precision and hits reasoning-heavy work first. There is usually a point past which the saving is not worth what it costs in reliability.
MythQuantisation is only about speed.
RealityMemory footprint is frequently the more important effect, since it determines what hardware a model fits on and therefore what a deployment costs.
Frequently asked questions
How much quality do I lose by quantising?
It depends on the precision level and on the task, and the honest answer is that you measure it rather than predict it. Run your own evaluation comparing the quantised model against full precision on representative work, making sure the set includes the hardest cases rather than only routine ones. Published comparisons are a useful starting point and do not transfer reliably to a specific workload.
Should I quantise or use a smaller model?
Try both and compare on your task, since they are different trades and the winner is not predictable. A quantised larger model and a full-precision smaller one can occupy similar memory while behaving quite differently — the quantised larger model often retains more capability on complex work, while the smaller one may be more consistent. Only an evaluation on your own workload distinguishes them.
Does quantisation affect the context window?
Not the supported length, but it does affect the memory available for handling long contexts. Since the memory required to process a request grows with context length, freeing space by quantising weights can make longer contexts practical on given hardware. The two interact through the same memory budget, which is why long-context workloads are often where quantisation pays most.