Quantization

A model’s weights are normally 16-bit floats. Quantizing to 8-bit or 4-bit reduces the memory footprint roughly proportionally, which means a model that needed multiple GPUs might fit on one, and throughput per GPU goes up. For self-hosted inference, that is the difference between viable unit economics and not.

Quality degrades gracefully rather than falling off a cliff: 8-bit is usually indistinguishable on most tasks, 4-bit is often fine for straightforward work and noticeably weaker on hard reasoning. The honest answer for any specific workload comes from an eval, not a blog post.

Quantization only matters if you run the weights yourself. If you call a hosted API, the provider has already made these choices for you — your cost levers are routing, caching, and token discipline instead.