Self-hosted LLM
Last verified: June 2026· definition
Self-hosting runs an open-weight model on your own cloud or on-prem hardware, giving you data residency and control at the cost of running and scaling inference yourself.
It’s the answer when regulation or data sensitivity closes the hosted-API path, or when very high volume makes fixed infrastructure cheaper than per-token pricing. Open-weight frontier models are strong, though the absolute quality ceiling is still set by the top hosted models.
A complete private stack usually includes inference serving, RAG pipelines over your data, optional fine-tuning, and the observability and cost controls you’d expect from any production system.
The compliance case and the cost case are different projects
Self-hosting for data residency is a requirement with a pass/fail test, and cost is whatever it costs. Self-hosting to save money is an economic argument that only holds above sustained high utilisation, because you pay for GPUs whether requests arrive or not. Conflating the two produces a deployment sized for the wrong goal — typically one provisioned for peak capacity to satisfy a compliance requirement, then judged on a cost comparison it was never going to win.
Operations is the recurring cost people omit
The visible number is GPU spend. The one that determines whether this works is the ongoing engineering: patching, model upgrades, capacity planning, monitoring, and on-call for an inference service that is now on your critical path. Comparisons that count only infrastructure reliably favour self-hosting, which is precisely why incomplete ones are common. A team without platform capability to absorb this generally should not take it on regardless of the arithmetic.
Quality varies by task, not uniformly
Open-weight models have closed much of the gap, but unevenly. On many bounded tasks — classification, extraction, summarisation, routing — a good open-weight model is genuinely competitive. On the hardest reasoning and long-horizon agentic work a gap generally remains. Because it is task-specific, public benchmarks answer the question poorly; evaluating candidates against your own workload before committing to hardware is what makes the decision sound.
Common misconceptions
MythSelf-hosting is cheaper than an API.
RealityOnly above sustained high utilisation, and only once operational staffing is counted. Below that threshold an API is usually cheaper as well as simpler.
MythSelf-hosting is the only way to keep data private.
RealityZero-retention agreements and appropriate contractual terms satisfy many privacy requirements. Self-hosting is necessary when data genuinely may not leave your network or jurisdiction, which is a narrower condition than it is often treated as.
MythOpen-weight models are broadly behind frontier models.
RealityThe gap is task-dependent. On many bounded production tasks they are competitive; on the hardest reasoning work a difference remains. Your own evaluation settles it for your workload.
Frequently asked questions
When does self-hosting actually make financial sense?
When utilisation is high and steady enough that the GPUs are meaningfully busy, and you already have platform capability to operate the service. The economics turn on utilisation because the cost is fixed while API cost scales with usage — idle capacity is the expensive failure mode. Model the total including operational staffing before deciding, since that term frequently changes the conclusion.
What hardware do I need to self-host an LLM?
It depends primarily on model size, precision, and your concurrency requirements, so the sizing exercise should follow model selection rather than precede it. Quantisation changes the answer substantially by reducing memory footprint at a measurable quality cost. The right sequence is evaluate models against your workload, decide which meets your quality bar, then size hardware for that model at your expected concurrency — reversing it means buying for a model you may not end up using.
Can I mix self-hosted and API models?
Yes, and it is often the strongest architecture. Routing bounded, high-volume, or residency-constrained work to a self-hosted model while sending the hardest requests to an API captures most of the cost and compliance benefit without needing the self-hosted model to handle everything. It also gives you a fallback path if the self-hosted service degrades, which pure self-hosting does not.