Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation (arxiv.org)

152 pointsby theanonymousone9 hours ago33 comments

pinyen_cheng 6 hours ago

That's pretty innovative. But a brand new tokenizer and training method need a lot of effort to do the retraining. Whether it will be widely implemented depends on real frontier performance it can reach.

alansaber 4 hours ago

Exactly. I used to think new tokenizers for domain-specific models was the way (makes intuitive sense, better representations etc), but the added overhead of essentially training from scratch made it much more expensive than a FT.

OOTY00TYOO 4 hours ago

If we cant' build the hardware fast enough. Can someone explain to me why there's apparently 50% of the gpu and memory that hyperscalers buy sitting in warehouses due to lack of data centers? The hyperscalers are hoarding tech and can't even use it. It's ridiculous.

IanCal 4 hours ago

That depends on how long it’s sitting there, given they are two different things to manage it makes sense that you’d either have gpus sitting unused while a dc gets built or a dc sitting empty while you wait for gpus.

idiotsecant 4 hours ago

Because slowing down compute purchase is collapsing the hype train which is the same as death. Vibes are a wave that must be maintained or crash down onto reality.

dormento 3 hours ago

Also, tinfoil alert, if they buy it means you can't (or are eventually priced out of the hw market). Their only "moat" (if you can call it that..) is the price of local inference capacity, which is still kinda prohibitive for the 99%.

alansaber 4 hours ago

They could probably rush them to production, but it's added logistics and would lead to the parts degrading faster.

dsjoerg 3 hours ago

I dont understand how this comment connects to the linked research paper. Anyone?

rhdunn 3 hours ago

There are reports and rumours that NVIDIA are forcing the data centre companies to buy the GPUs upfront. Though there isn't a lot of clear evidence that this is the case.

[1] https://tech.yahoo.com/gaming/articles/did-nvidia-manufactur...

[2] https://www.reddit.com/r/Games/comments/1xdjbd1/multiple_rep...

[3] https://www.techradar.com/computing/gpu/is-the-gpu-stock-cri...

[4] https://www.businessinsider.com/amazon-nvidia-ai-gpu-data-ce...

Lerc 2 hours ago

As far as I can tell, because that is not happening.

The suggestion seems to have come from the claim that to maintain a priority queue, you have to be buying consistently. That might mean buying more than you need to ensure supply when you need it. The assumption that the excess goes into a warehouse seems to have come from someone's fevered imagination.

If there are GPUs built and not yet installed, and people who want to be using them now, money can be made getting the supply to find the demand. Most large scale companies know how to do that sort of thing. If someone has a data center and no GPUs arriving for six months, paying a few million to trade the GPUs that will exist in six months for the ones that exist now gets them an additional six months of operation, while ensuring the original buyer will have GPUs when they need them, plus a few million.

torginus an hour ago

Yeah. Don't know anything about data center GPUs, but it's common knowledge you can get like 80% of the performance of a gaming GPU at half the power or even less. That's basically how Nvidia's mobile GPUs are. So if you had a warehouse full of GPUs, you could just go ahead and install them, run them at low power, and still get like 2/3rds of the performance.

owebmaster an hour ago

Lack of customers*. SpaceXaitwitter has some big beautiful idle datacenters with all GPUs ready to use.

swiftcoder 8 hours ago

Who's going to train a nice big coding model based on this? I don't have the GPUs to do it...

LoganDark 7 hours ago

I hope Apple Silicon will one day be competitive with GPUs. It dominates much of the CPU market, and it has a GPU, but it's still not even remotely comparable to Nvidia for inference (factor of >5 in some cases, especially in prefill) or even for gaming ("edge computing" in Nvidia terms).

I feel like their GPU architecture is going to need a significant redesign. There are limits to this tile-based mobile architecture.

mycall 2 hours ago

Depends what you are comparing. Apple Macbook Pro M5Max out performs NVIDIA RTX N1X laptops for about the same cost.

LoganDark an hour ago

Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090.

Not comparing price or efficiency or value or anything like that. Just the platforms as a whole.

swiftcoder an hour ago

> Fastest consumer Apple silicon versus fastest consumer Nvidia. So M5 Ultra vs RTX 5090. Not comparing price or efficiency or value or anything like that. Just the platforms as a whole. So M5 Ultra vs RTX 5090

The problem here is that they aren't equivalent platforms - no amount of money will buy you an RTX 5090 with 512 GB of RAM.

If you want to buy the 16x RTX 5090s you need to get to the same 512 GB of RAM, we're talking $100,000 just in GPUs, industrial levels of power/cooling/noise, and the interconnect bandwidth is still going to kill it for normal inferencing workloads.

Of course, that 16x cluster is an absolute beast in terms of raw compute - it's just not all that useful for the things people are trying to do with such machines today.

siamaksalman 8 hours ago

I'm trying to run it, so far no luck

cubefox 4 hours ago

This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.

However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.

nickhodge 8 hours ago

https://www.youtube.com/watch?v=nv5_PH2uPVI

imtringued 7 hours ago

I keep getting clickbait youtube videos about how "NCM" obsoletes "LLMs" and "Nvidia memory chips" (yes, it's that bad).

This makes me wonder what these people think an "LLM" even is. The name itself implies that it is large and about language. It does not imply that an "LLM" is inherently autoregressive or that it has any particular architecture even though most LLMs are transformers. Anything can be a large language model as long as it is large and models language.

NCM is basically a hierarchical diffusion transformer where the further layers take one big token and then generate two smaller tokens. So it is more like an evaluation strategy. You could apply this to any model architecture.

I personally think the name NCM itself is terrible because it contains the word "next" which inherently implies an ordering and somehow we are supposed to think that "LLM" which contains no mention of ordering, is constrained to ordered token production.

hsuduebc2 5 hours ago

I think they think an LLM is just whatever another LLM tells them it is when they ask it to generate a script.

alansaber 4 hours ago

Hey, me too.

totetsu 7 hours ago

Can we apply any insights from image and video compression and partitioning to this kind of dimensional masking ?

shiandow 6 hours ago

I'm not sure why but using X instead of the word "times" annoys me. Probably because one of the first things I got taught at university is that mathematical notation is not prose and shouldn't be used as such.

spiderfarmer 6 hours ago

But what about “got taught” instead of “was taught” and the missing comma before “but”?

shiandow 6 hours ago

Well they didn't teach me English. Or in English for that matter.

And I don't think I've ever used the Oxford comma.

There was probably some rule about not beginning sentences with Or and And but you probably know better than I.

jabron 5 hours ago

The Oxford comma is the comma before the conjunction in a list of three or more terms, e.g. "one, two, and three". I think your sentence actually didn't need a comma anway.

wccrawford 3 hours ago

Different cultures have different norms and rules. I think you're saying that, in your native language, they taught you not to use mathematical notation in prose.

They didn't teach you that about English, though. It's quite common to write "21x" for "21 times" in English and is pretty much universally understood and accepted by native speakers.

shiandow 3 hours ago

It's a more general rule about when and where to use mathematical notation. Though style rules do differ in places.

For what it's worth I've not seen "12x" in formal contexts, but if that is proper then the times symbol still shouldn't be styled as part of the maths formula imho.

sorenjan 2 hours ago

The first time I heard anyone pronounce 2X as "two ex" instead of "two times" was Steve Jobs in one of his presentation.

cubefox 5 hours ago

I wonder whether "10" in "10×" also counts as mathematical notation. Or if just "ten times" is allowed.

shiandow 3 hours ago

Technically yes, but quoting mathematical constants is definitely allowed. You could also say O(N) times for instance. The kind of language that is discouraged is writing "therefore efficiency increases by O(log(N)) times" as "∴ efficiency increases by O(log(N)) *".

For one it simply looks like you forgot to finish the formula.