Self-Hosted LLM vs API: A Cost Breakdown
Self-hosting an open-weight model or calling a hosted API? The drivers on each side, the break-even point, and the hidden costs.
Self-hosting an open-weight model or calling a hosted API? The drivers on each side, the break-even point, and the hidden costs.
We chose to run our own AI infrastructure instead of building on closed APIs. Here is the case for owning your stack, and the honest cost of doing it.
How much RAM, VRAM, and disk you need to self-host a 7B, 13B, or 70B open-weight model, with cited memory figures and the speed you can expect.
A walkthrough of how to build semantic search with embeddings using RAG AI to index and chat with your docs, and when keyword search still wins.
How to turn scanned images and PDFs into typed fields your app can store, using local OCR and a language model, with the validation that keeps poor quality out.
The self-hosted LLM vs API cost question comes down to one thing: how much you use it, and how steadily. Calling a hosted model API costs you a few dollars per million tokens with zero fixed cost, so it wins when your usage is low or bursty. Self-hosting an open-weight model means renting a GPU by the hour or month, a fixed bill whether you use it or not, so it wins only at high, steady volume that keeps the hardware busy. Most teams starting out are firmly in API territory and do not need their own GPU yet.
The honest version is that there is a crossover point, often called the break-even, where the fixed cost of a GPU finally beats the per-token cost of an API. Where that point sits depends heavily on your volume, your model, and how well you keep the hardware utilized. This guide walks through the cost drivers on each side, the hidden costs nobody quotes you, and how to find your own break-even. If you want the hardware side in detail, see our guide on VPS specs to run an LLM, and for picking a server in general, how to choose a VPS. The pattern here mirrors the one in which hosting type is best for small websites: match the tool to where you are now, not where you might be someday.
Hosted APIs charge per token, with no fixed monthly cost. You pay separately for input (your prompt) and output (the model's reply), priced per million tokens. A token is roughly three-quarters of a word, so a million tokens is a long afternoon of chat or a few hundred document summaries. You pay only for what you send and receive.
Prices span a wide range by model. On the closed-model side, OpenAI's flagship GPT-5.5 lists at $5.00 per million input tokens and $30.00 per million output tokens, while its smallest model, GPT-5.4-nano, is $0.20 input and $1.25 output (OpenAI API pricing, figures approximate and current as of June 2026). Open-weight models served through a provider are cheaper still. Together AI lists Llama 3.1 8B at $0.18 per million tokens and Llama 3.3 70B at $0.88 per million (). All of these are approximate and change often, so treat them as a snapshot, not a contract.
We built a tool that scores titles and headlines with an LLM. Here is how to make the model grade consistently, hold a strict format, and stop it refusing.
The structure is what matters more than the exact numbers. The cost scales linearly with use: ten times the traffic, ten times the bill. There is nothing to provision, no idle waste, and no maintenance. Output tokens usually cost several times more than input tokens, so chatty, long-form responses cost disproportionately more than short ones. That linear, no-floor shape is the whole appeal, and also the whole limitation.
Self-hosting flips the structure: you pay a fixed price for a GPU and the per-token cost effectively goes to zero. You rent (or buy) a graphics card, load an open-weight model like Llama or Mistral onto it, and serve requests yourself. The meter runs on time, not tokens, so the bill is the same whether the card is flat out or sitting idle.
Cloud GPU rental is the usual on-ramp because it avoids a hardware purchase. As a snapshot of current marketplace rates, RunPod lists an RTX 4090 at $0.69 per hour, an A100 PCIe at $1.39 per hour, and an H100 SXM at $3.29 per hour (RunPod pricing). Spot and community tiers run cheaper, dedicated enterprise instances run higher, and a small model can run on a single consumer card while a 70B model wants a high-memory data-center GPU. Run an H100 around the clock and the math is simple: roughly $3.29 an hour is about $2,400 a month, paid in full whether you serve one request or a million.
The cost drivers on the self-hosting side are the GPU class you need (driven by model size and required speed), how many hours you keep it running, and your utilization. The card does not care whether it is busy. A GPU at 10 percent load costs about ten times as much per token as the same card at full load (Braincuber). That single fact is why self-hosting punishes low and bursty traffic so badly.
The break-even is the monthly volume at which a fixed GPU bill becomes cheaper than per-token API fees. Below it, the API is cheaper because you pay nothing for idle time. Above it, self-hosting wins because the per-token cost has fallen toward zero. For a mid-size open model, that crossover tends to land in the range of hundreds of millions of tokens per day.
One published analysis puts a concrete number on it. Running Llama 70B, self-hosting costs roughly $4,360 a month at around 500 million tokens per day, against roughly $22,500 a month to serve the same volume through an API, a fivefold gap in self-hosting's favor at that scale. The same piece pegs the practical crossover near $4,200 a month of API spend (Braincuber). The shape of the curve is the lesson: API cost rises in a straight line with usage, while self-hosting is a flat line set by your GPU. The two cross at one point, and that point moves with your model and your prices.
A few things shift the break-even, sometimes dramatically:
Both sides have costs that never appear on the headline price, and ignoring them is how budgets blow up. On the API side it is mostly per-token sprawl and data movement. On the self-hosting side it is idle time and the engineering effort to run the thing. The honest accounting includes all of it, not the sticker price.
On the API side, watch for:
On the self-hosting side, watch for:
Each option wins in a clear, predictable zone, and the deciding factor is your sustained volume and how steady it is. Low or bursty usage favors APIs, because you pay nothing for idle time and skip all the operational work. High, steady, predictable volume favors self-hosting, because a busy GPU drives the per-token cost toward zero. Most teams sit in the first zone longer than they expect.
Choose a hosted API if:
Choose to self-host if:
A reserved or dedicated GPU is the right move when you have consistent, heavy, predictable inference and the team to run it. For occasional or unpredictable workloads, you genuinely do not need it yet, and forcing it on bursty traffic is one of the more expensive mistakes in this space. Platforms that host open models, including Carpathian, can shift some of that operational burden by running the hardware for you, though that convenience is itself a cost to weigh against doing it yourself.
The honest limitation is that there is no universal break-even number, and anyone who quotes you one without your inputs is guessing. The crossover depends heavily on three things you control: your token volume, the specific model you run, and how well you keep the hardware utilized. Change any one and the answer moves, sometimes by an order of magnitude.
The published figures here, like a roughly $4,200-a-month crossover or a 500-million-token-a-day break-even, are useful reference points from a specific analysis, not laws (Braincuber). Your own number could be far lower if you are escaping an expensive flagship model, or far higher if you are competing with a cheap open-model provider running at scale. The only reliable method is to do the arithmetic with your numbers: estimate your monthly tokens, multiply by current API rates for your model, and compare that against the monthly cost of a GPU sized for your model at your expected utilization.
Work through it in order and the answer falls out. The point is to compare your API spend at your volume against the fixed cost of a GPU that could do the same job, with utilization factored in honestly. It takes five minutes and beats any rule of thumb.
If the two paths come out close, default to the API. It carries no operational burden and no idle risk, and it lets you switch later once your volume is high and steady enough to make self-hosting an easy call. The better-run infrastructure tends to be the kind you stop thinking about: sized to what you use, quietly dependable, and priced for the work it does rather than the capacity it might.
