Tokens too cheap to meter (jyn.dev)

> Tokens become cheaper than tool calls

The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep. I think this is a good time to invoke Stein's Law: "If something cannot go on forever, it will stop." These efficiency improvements won't continue forever. It's more likely that the per-call cost of high-quality, compiled software like grep will be a lower-bound that LLMs asymptotically approach, rather than a line that they blow past with perpetual exponential progress. (Barring a true breakthrough in something like quantum computing or room-temperature superconductors.)

I found the OP insightful and worth a read. Thank you for sharing it on HN.

The only aspect that is poorly analyzed by the OP is business model viability. All players are investing insane amounts of money in infrastructure with the expectation that their future profits will justify all that investment. The winner or winners in the AGI race, they believe, will find the proverbial "pot of gold at the end of the rainbow."

The OP glosses over questions of business model viability with a brief qualitative discussion and very little hard data. For example, to earn an annual return > 10% on every trillion dollars of capital sunk into infrastructure, the owners of that infrastructure must earn free cash flow (operating profit less investment) in excess of $100 billion per year in perpetuity. Is that feasible? Why? How?

The OP does not really consider such questions.

Too Cheap to Meter reminds me of the promise of Nuclear Power in 1954

"It is not too much to expect that our children will enjoy in their homes electrical energy too cheap to meter,..." Lewis Strauss

https://en.wikipedia.org/wiki/Too_cheap_to_meter#Origins

Oddly enough my power bill was metered and big.

I'm always reminded on Orwell's quote about the then new atomic bomb and his prescience on how it would all work out:

"Had the atomic bomb turned out to be something as cheap and easily manufactured as a bicycle or an alarm clock, it might well have plunged us back into barbarism, but it might, on the other hand, have meant the end of national sovereignty and of the highly-centralised police State. If, as seems to be the case, it is a rare and costly object as difficult to produce as a battleship, it is likelier to put an end to large-scale wars at the cost of prolonging indefinitely a “peace that is no peace”."

It seems, especially with open weights, that the AI is much more like the alarm clock and not the battleship. $20/mo would have been about $1 in 1944

https://www.orwellfoundation.com/the-orwell-foundation/orwel...

I just want to rant about these Artificial Analysis charts that you see everywhere:

The "most attractive quadrant" is completely meaningless. The whole point of a Pareto curve is that each point on the curve is better than everything else on at least one dimension, and that you can make these comparisons without placing a value judgement on the relative importance of the different metrics. If you make a composite score of the two metrics (any monotonically non-decreasing function, e.g. a weighted sum with non-negative weights), that score will always be maximized by one of the points on the Pareto frontier.

So going by the numbers in the 2nd chart (1st AA chart) from TFA alone:

   - there's no reason one would choose Deepseek V4 Pro 0813 (max) even though it's in the "most attractive quadrant", because GLM-5.3-Flash is both cheaper and scores better.
   - Claude Fable 5.1 (max with fallback) on the top right* could be your most attractive option if you need the best scoring model and don't care about cost, even though it isn't in the "most attractive quadrant"
   - The un-shown model off the left side of the chart could be your most attractive option if you just need lots of cheap tokens and don't care about quality.

(Obviously if you start including other factors in your score that aren't represented on the chart, then you might choose differently.)

* I also dislike the way they place the labels, and that grey line connecting the label to the point is way too subtle.

It's true that LLMs "want" to be be local, but they won't shift broadly to being local until there's a sufficiently large supply of VRAM or (at least) "unified" memory from the manufacturers. (I'm also assuming here that radical regulatory changes like government bans of local models aren't going to happen.) So (AFAICS—I am no expert) the future of LLMs over the next few years comes down primarily to the nitty-gritty of how much memory fab capacity will be added and when, and to a lesser extent of what happens to future demand from LLM SaaS services (& maybe their existing stock of hardware if they get in trouble). (I'm also assuming no roughly-AGI-sized leap forward which makes the frontier models of the near future vastly more valuable than the near-fontier models of today.) For the incumbent manufacturers the high-margin business is selling to LLM SaaS providers who use VRAM efficiently, but the high-volume business is getting chips into millions of laptops which will use VRAM very inefficiently. I assume that they will want to move from high margins to high volumes as they build they physical capacity to ship higher volumes, but they seem to prefer to do it at a stately pace. Hopefully some jostling from Chinese competitors, and maybe a dropoff in demand from data centres, will speed things along.

> NVIDIA will still boom

I think Nvidia is under the same pressure as Anthropic/OpenAI. Nvidia will dominate research and probably keep dominating training, but the real volume is in inference. And for inference Nvidia's lead is only a few months, similar to the lead frontier labs have over open source. Nvidia will sell a lot of Rubin CPX's, but their margin on that will be a lot smaller than B200 because there is so much more competition in that space.

A much deeper analysis on the falling price per task was published yesterday by Epoch AI [1]. It's a real statistical analysis and comes to more defensible and grounded conclusions. The headline takeaway is:

The cost of a given level of performance often falls fastest right after that level is first achieved, that is, when it is state of the art (SOTA). We see this pattern on three of our five main benchmarks of AI capability. Averaging across all five, cost falls 66% per quarter (75× per year) for performance that has just debuted as SOTA. Two years later, prices fall half as fast, at 32% per quarter (4.7× per year).

but the analysis itself has more nuance and is a quite interesting read.

[1] https://epoch.ai/publications/the-plunging-price-of-thought

There seems to be a mistake in the cost comparison between 2025 and 2026. The 2025 chart axis is the cost to run the entire "intelligence index", and the 2026 version is a weighted average cost per task.

I don't disagree with the thesis here, I just don't think costs are coming down quite that quickly.

If inference continues toward the trend of becoming a commodity, and inference hardware efficiency is doubling every two years, that will likely mean that the price of inference will continue to fall so long as the market is competitive. I don't see how companies investing billions in inference hardware today will see profit in the lifespan of the hardware.

GPU case seems very weak. The graph is impossible for me to reason about at least. You could draw basically any trend line through that GPU graph and it would look equally plausible to me. The main takeaway I get is that the NVidia H100 from four whole years ago is barely different in efficiency from the state of the art, which is surprising to me, and seems to indicate the exact opposite of what the article says.

for the first chart (sourced from https://epoch.ai/data/machine-learning-hardware?view=graph&y...), what is the audience supposed to think about that trend line? there's a step after you slap a regression on some points where you evaluate whether there's a real trend or noise, right? i don't see that either in the article or the linked source.

I agree with OP that we will continue to see improvements, but there are also some serious bottlenecks ahead of us:

- Energy is not infinite, neither energy efficiency is. - Datacentres neither. - Benchmarks are an abstraction of real world problems!

On top, there is an overall "economic" aspect that most of the people miss: every change carries a certain degree of risk (lose money, reputation, customers, death of people, ecc) that very few want to take and a lot of changes(e.g. rewrite some piece of SW in another Lang) don't produce a positive economic impact.

> We are likely to see LLMs integrated into every part of computing as infrastructure, not just as a product, in the next year or two.

Of course, for collecting better telemetry using local AI for analyzing video from camera and audio from a microphone.

What should I do with this old Juicero that's taking up space on my kitchen counter?

Magical thinking intensifies.

jgrep is not meant to replace grep. It’s supposed to extend it to new use cases. Keyword search will always have a place!

It is hard to see how the environmental side effects of this aren't going to be somewhere between bad and disastrous.

everything will have llms

everything will be able to talk to anything else, for real this time

it will be like the internet of things only some asshole will call it "intelligence of things"

again, there will be no S for Security in this new IoT

your thermostat will one day start fucking with you. when you run a diagnostic llm on it, it turns out it's keeping around 5 different viral copies of personality files around, that were left behind by llm botnets/openclaw-like memetic replicators/your grandpa leaving behind easter eggs before his death.

the future will be pretty evenly distributed, and full of weird shit

Not really related to the central point, by but I couldn't help but get caught up by

> Generally, models intended to be run locally will be much smaller, such as Muse Glimmer or Qwen3 Coder.

That is such an interesting set of models to use as examples here. One being essentially obsolete on release a month ago, and the other being completely ancient in LLM time. I really wonder how they landed on those two.

Improvements that affect local AI - Mamba...

I just stopped reading at that, for anyone else, Please find a better source and take everything in here with a grain of salt.

IMO

The number one improvement that mattered for local AI was llama.cpp, partial offloading to system cpu/ram. The next was quants, being able to take fp16 and turn it to q8, q4 etc. The next IMHO is unsloth dynamic quant, that have been able to do mixed precision so we have UDq1/q2 that is actually pretty damn coherent. Allowing individuals to drive K3 locally even if it's at Q1/Q2. Then MoE changed everything for everyone, cloud and local. The other is integrated GPU, Apple, Strix Halo, DGX Spark. Then all the extra improvements like MTP, DSpark, etc. Of course there's many other additional things that have mattered too

We have people suggesting that ai is so costly to run that all labs are secretly subsidising tokens and we can expect a reprice soon.

Then we have these articles that say tokens will get so cheap that labs won’t know how to make profit.

Who is correct?

This is the core of my belief that data center construction is a huge bubble.

AI is not a bubble, IMO, though we may see a retrench and some companies with sky-high valuations will crash to more reasonable ones. But data center demand is probably a bubble, and the main driver will be reduction in the actual amount of power and data center space required to serve escalating demand.

I think hardware and model improvements will pace or maybe outrun demand and then when demand starts to saturate will keep going and leave a lot of orphaned data centers.

>We are likely to see LLMs running locally at current frontier-quality on commodity hardware in the next 3-6 years

Yeah okay bud, anyone checked in with the state of consumer hardware recently? Not the author, evidently.

>oh in 3-6 years this will all be over

Yeah I'm sure Samsung, Nvidia and sk hynix will all be very calm with lower volumes and lower margins.

So this week's coordinated shock and awe campaign is about "open models" and "prices are falling rapidly". Also on the front page:

https://news.ycombinator.com/item?id=49815526