Why Efficient Inference Beats Brute Force
I am convinced the future of useful AI is small, efficient, often task-specific models, not an endless race for the biggest one. Quality per watt beats brute force.
I am convinced the future of useful AI is small, efficient, often task-specific models, not an endless race for the biggest one. Quality per watt beats brute force.
We chose to run our own AI infrastructure instead of building on closed APIs. Here is the case for owning your stack, and the honest cost of doing it.
For a model that must fit on a small device, the parameter budget is fixed. The place to spend is inference-time compute.
We throw away a staggering amount of perfectly useful compute. Here is why a lot of AI and cloud work can run on older, upcycled hardware, and where it honestly cannot.
We ship megabytes of script to display a paragraph of text, and we call it progress. I think we forgot that efficiency was ever a virtue.
Remember when "better AI" just meant a bigger number? More parameters, more GPUs, more megawatts, and somewhere over the horizon, the model that finally does everything. We are still being sold that horizon. And I am tired of it.
I run an infrastructure company. I watch the power bills. I watch the GPU spot prices. And I have spent enough time staring at benchmark tables to believe something that sounds almost heretical right now: for most of the work people genuinely want done, small language models are not the consolation prize. They are the point.
Let me be clear before anyone reaches for the "AI skeptic" label. I am not skeptical of AI. I am skeptical of the way we have decided to BUILD it. Those are different arguments, and conflating them is how the industry dodges the hard questions.
For a few years, scale was a fair proxy for quality. Make the model bigger, feed it more, and it got smarter in ways you could measure. That was a thing. I am not going to pretend it wasn't.
But proxies rot. Somewhere along the line, "bigger" stopped meaning "better for you" and started meaning "better for the press release." The model that needs a rack of accelerators to answer a question about your calendar is not impressive. It is a forklift delivering a sandwich.
Here is the part nobody on the keynote stage wants to dwell on. The curve is bending. Each new tier of scale costs exponentially more, in compute, in capital, in energy, and returns less than the last. Diminishing returns is not a slur. It is arithmetic. And when you are spending nation-state quantities of electricity to shave a couple of points off a benchmark almost no one will ever notice, you have to ask the uncomfortable question: who is this for?
Let me give you the anchor that made me stop hedging.
Microsoft's Phi-3 technical report describes phi-3-mini, a 3.8 billion parameter model, scoring 69% on MMLU and 8.38 on MT-bench. The authors say its overall performance "rivals that of models such as Mixtral 8x7B and GPT-3.5." A model small enough, in their words, to be deployed on a phone, going toe to toe with models many times its size.
Sit with that for a second. The headline race has been about trillion-parameter behemoths in distant data centers. Meanwhile a model that fits in your pocket is landing in the same neighborhood as the giants on the benchmarks that supposedly justify all that scale. The brute-force premium, the thousandfold jump in size, buys you a sliver. Sometimes it buys you nothing you would feel.
This is not a one-off fluke from one lab. It is a pattern. Careful data, better training, and task focus keep closing the gap that raw parameter count was supposed to defend. The wall the giants are paying so dearly to climb has a staircase next to it.
Free AI is not a gift. Someone pays for every query, in energy, in data, in dependency. Here is who, and why I'd rather pay honestly for compute I control
I want to retire "how big is it" and replace it with two questions I cannot stop thinking about. How much useful work does this model do per watt? And per dollar?
Because that is the metric your problem feels, even if the marketing never mentions it. A model is not abstract. It runs on a machine that draws power, occupies a rack, throws off heat, and drinks water through a cooling loop. Every inference has a physical cost, and at the scale these things are being deployed, that cost is not a rounding error. It is a power-grid line item. It is a county's worth of cooling water. Treating it as a footnote is how we ended up building cathedrals to answer trivia.
Don't get me wrong, the raw numbers still matter for some things. But the model that solves YOUR problem at a tenth of the energy is not the lesser model. By the only score that survives contact with a utility bill, it is the better one.
Here is the line I refuse to cross, because crossing it would make me exactly the kind of reflexive critic I cannot stand. Frontier-scale models matter. They genuinely do.
When you are pushing into territory nobody has mapped, the biggest, most general models are the instrument. Drug discovery. Protein structure. Climate modeling at scales no human team could hold in their head. Frontier research that needs the broadest possible reasoning over the messiest possible inputs. That work is hard, it is worth doing, and the giants earn their keep there. I will defend that all day.
But that is the workshop. That is the precision instrument you wheel out for the one job that demands it. You do not keep it running 24/7 to summarize an email. We have taken a tool built for the frontier and shoved it into a million mundane tasks it is wildly overqualified for, then acted surprised that the bill, and the carbon, came due. A frontier model answering your support tickets is a solution searching for a problem it already left behind.
This is where the frustration turns into something I am building toward, because the case for small models is not really about being against bigness. It is about everything efficiency lets you have.
Private. A model small enough to run on hardware you control is a model whose data never leaves the building. No round trip to someone else's data center. No fine print about how your prompts might be used. For anyone touching health records, legal files, or anything they are responsible for protecting, that is not a nice-to-have. It is the entire ballgame.
Local. Small models run at the edge: on your laptop, your phone, a modest server in your own rack. That means no latency tax waiting on a distant cluster, and it keeps working when the network does not. The intelligence lives where the work is.
Affordable. When a capable model fits on hardware a small team can afford, capability stops being a thing you rent by the token from whoever owns the biggest cluster. A clinic, a school, a two-person shop, a researcher in a country the hyperscalers have not bothered to wire up, all of them can run useful AI without a hyperscaler's permission or invoice. That is the part that moves me.
And efficiency does not stop at picking a smaller model. Quantization, the technique of squeezing a model into lower-precision math so it runs leaner, has gotten good enough that careful 4-bit methods can land within a couple of percent of full precision on many tasks (arXiv study on quantization strategies). You are not throwing the model away to make it fit. You are getting most of the capability at a fraction of the footprint. That is engineering doing what engineering is supposed to do.
I am not interested in writing a complaint and calling it a contribution. So here is the work.
At Carpathian we host AI models on infrastructure we build and run ourselves, and the whole design philosophy is efficiency: get the most useful inference out of the least power and the least hardware, and pass that down as pricing you can predict. The interesting questions to me are not "how do we rent the biggest GPU" but "how small can a model be and still nail this specific job," and "how much older hardware can we upcycle instead of sending to a landfill." That second one matters more than the industry admits. The greenest accelerator is frequently the one already built.
This is the part I will hedge, because I would rather be honest than sound certain. I do not think small models win every fight, and I am not pretending efficiency is a clean, solved problem. It is messy. It is task by task. Some workloads will always reach for the giant, and they should. But the default has been backwards. We reach for brute force first and ask whether we needed it never.
So flip it. Reach for the smallest thing that does the job, and only scale up when the work demands it and you can prove it does. That is not a limitation. That is craft.
The future of useful AI is not the biggest model in the world. It is the right-sized one running quietly on hardware you can afford, doing the work in front of you and asking the grid for almost nothing.
