The Hidden Energy Cost of AI, and What Low-Power Inference Could Look Like
The AI energy cost is treated as an inconsequential while the power demand is at the size of some countries and why efficiency, not scale, is the future.
The AI energy cost is treated as an inconsequential while the power demand is at the size of some countries and why efficiency, not scale, is the future.
Years of building infrastructure taught me that trust is earned quietly, honesty about limits beats big promises, and locking people in is a slow death.
On saturated consumer hardware, you do not buy small-model quality with a faster framework. You buy it with better data and a better training objective, with numbers to back it.
Speculative decoding and multi-token prediction cut byte-level generation latency without touching output quality. Here is what we measured on Veritate's models
Below roughly four bits per weight, post-hoc QAT on byte-level models is catastrophic, and quantization-aware training recovers nearly all of the lost quality
For a model that must fit on a small device, the parameter budget is fixed. The place to spend is inference-time compute.
When you send a prompt to a large model today, the work does not happen on your laptop. It happens in a building you will never see, on a rack of accelerators pulling more power than your whole street, cooled by water that came from somewhere with a name. The AI energy cost is not theoretical and it is not small. It is just hidden, by design, behind an interface so smooth that the only friction you feel is the half-second before the answer appears. And that smoothness is the problem. We have built an industry that hides its own metabolism.
Here is the figure I cannot get out of my head. The International Energy Agency estimates that data centers consumed around 415 terawatt-hours of electricity in 2024, roughly 1.5 percent of the world's power, and projects that to roughly double to about 945 terawatt-hours by 2030 (IEA, Energy and AI). The agency's own framing is the part that should stop you cold: by 2030 these buildings will draw as much electricity as the entire country of Japan does today (IEA news release). Japan. Not a city. A G7 nation.
And AI is the engine of it. The IEA expects demand from AI-optimized data centers to more than quadruple over the same window. Read that twice. The fastest-growing line item in global electricity is not heating, not transport, not heavy industry. It is the thing that writes your emails and summarizes your meetings.
Then there is the water, which is the part hardly anyone priced in at all. Researchers at UC Riverside found that training a single model, GPT-3, in Microsoft's US data centers evaporated roughly 700,000 liters of clean freshwater, and that something like twenty to fifty of your queries to a chatbot quietly drinks a 500ml bottle (UC Riverside). You did not see the bottle. That is the entire trick. The cost was moved somewhere you would never look, into a watershed in a place you have never been, and then it was rounded down to a footnote.
We throw away a staggering amount of perfectly useful compute. Here is why a lot of AI and cloud work can run on older, upcycled hardware, and where it honestly cannot.
I want to be careful here, because there is a lazy version of this argument and I refuse to write it. AI is not the problem. The hype is the problem. The waste is the problem.
Because some of this is worth every watt. A model that reads a retinal scan and catches diabetic retinopathy before a human radiologist would have, in a clinic that has no radiologist, is worth the power it draws. Protein structure prediction that compresses years of wet-lab work into an afternoon is worth it. Live captioning that lets a deaf student follow a lecture in time is worth it. These are not toys. These are the legitimate, load-bearing uses of the technology, and I will defend them against anyone.
What I will not defend is the other ninety percent. The chatbot bolted onto a thermostat. The "AI-powered" feature that is a regex with a marketing budget. The image generator burning a data center's worth of cooling water so a brand can skip a fifty-dollar stock photo. So much of what we are spending this energy on is a solution searching for a problem, and we are paying for the search in megawatts and aquifers. We took a tool that can read a tumor and we used it to autocomplete a Slack message that did not need to exist.
The grievance is not that AI uses energy. Everything worth doing uses energy. The grievance is that we stopped asking whether the thing was worth the energy at all.
The industry's reflex to every limit is the same: build more. More data centers, more gigawatts, a new reactor for every campus, a fresh substation, another claim on another river. Scale is the only lever the big players seem to know how to pull, and I understand why. Scale is fundable. Scale photographs well. A press release about a billion-dollar campus writes itself.
But scale is not progress. Scale is just the same inefficiency, larger.
Here is the thing that drives me a little mad as an engineer. We are pouring this much power into hardware that, much of the time, is barely being used well. Inference, the act of running an already-trained model to answer you, does not have to look the way it looks now. Models can be quantized, distilled, and pruned to a fraction of their footprint with very little loss in quality for the task at hand. Older hardware that the hyperscalers have written off can be upcycled instead of landfilled. Workloads can be scheduled to follow clean power and cool air instead of forcing the grid and the chillers to bend to a workload that never sleeps. None of this is exotic. Most of it exists. It is simply less glamorous than building a new monument to compute, so it does not get built.
Efficiency is the frontier we keep stepping over on our way to the groundbreaking ceremony.
So picture the other path for a minute, because I think about it constantly. Imagine inference infrastructure designed from the first principle that the cheapest watt is the one you never spend. Right-sized models that match the job instead of a single giant model brute-forcing every request, including the ones a smaller model would have answered identically. Hardware chosen and tuned for performance-per-watt rather than peak benchmark glory. A second life for servers that still have years of useful work in them, kept out of the e-waste stream where most of them go to die.
This is the work I am doing at Carpathian, and I will be honest that I am at the beginning of it, not the end. I am researching ultra-low-power data center designs and the upcycling of older hardware, and I am exploring what efficient inference looks like when you treat energy as a first-class constraint instead of an externality you can dump on a community downstream. We run our own US infrastructure rather than reselling a hyperscaler's, which means the efficiency is ours to win or lose, not someone else's to obscure. I am not going to pretend I have solved this. I have not. The honest claim is smaller and I think more useful: it is solvable, and the industry has decided not to try because trying does not scale a valuation.
That is the part that gives me hope, oddly. The waste is not a law of physics. It is a choice. And anything that is a choice can be chosen differently.
The reason any of this matters is that the cost never disappears. It only moves. Move it off the user's bill and onto the grid. Off the grid and onto a river. Off this decade and onto the next one. A hidden cost is still a cost, and someone downstream, a town with a strained water table, a grid operator staring at a demand curve that bends straight up, eventually pays it in full and without their consent.
I got into infrastructure because I love the machine. I love that you can take electricity and sand and turn it into something that answers a question. I am not here to tell you to stop using AI, and I would be a hypocrite if I did, because I build it for a living. I am here to tell you that the smooth interface is lying to you about what is happening on the other side of it, and that we can build the other side honestly if we decide to.