Training a 3.8B LLM to 0.384 CORE for $998 (hugovergnes.github.io)

72 pointsby Anon847 hours ago13 comments

brainless 5 hours ago

More and more such experiments. I felt sad for a couple months when I realized that writing code will not be the same since. Now I am on the other side.

LLMs are interesting in their own ways but as an engineer, this is a way to unlock a new way of building software.

I recently build a Claude-assisted Excel/CSV parser for a US based property management system (tax compliance). Uses Haiku and has a lot of deterministic code to extract column/row combinations to check known formats and finally handing out the headers to Haiku to give us a translation plan to our support columns.

These would eventually become part of the software, in a tiny LLM. The gap between training (such tiny LLMs) and inference will shrink. We can consult Claude for edge cases, create sample dataset and train a the tiny LLM on demand so we go to Claude less.

The tooling that a project needs is really important. Something I have been feeling as well. Not just in LLM building projects, but regular software projects that are LLM generated.

springtimesun 2 hours ago

I think I am on your arc as well. My learning on different topics is growing every day, but there’s a limit to how much I can absorb. With the LLMs the experiments stay just beyond that horizon and I keep chasing.

Stated too strongly, but I think this could be the model for education (some subjects anyway). Everything personalized to your learning goals, grounded in experiments that give a tight feedback loop and with a model that never gets tired of re-explaining something for the 10th time.

brainless an hour ago

Yes they are great for learning at own pace, trying out new things.

I have accepted two things that make me a happy engineer now: AGI is not here no matter what they say and LLMs are still very useful if one knows how to use them.

They are another layer of abstraction and like you said they do not tire. There is a lot of optimization needed so we can reduce wastage (running 1T+ LLMs for most work is wastage).

pulse7 an hour ago

I hope we will soon have an open-source project for training such small LLMs where one can just pick the architecture (like Qwen / DeepSeek / etc.), parameter count, dataset, ... and then let it run on a local/rented GPUs...

rao-v 2 hours ago

This is really neat! I'd be tempted to try this again targetting ~1B params and the entire cookbook of "current" small model ideas: gated delta nets (or is ~2K context too short to benefit?), per layer or n-gram embeddings, gated residuals etc.

johnnylambada 5 hours ago

I don’t blame you for using an LLM to write an article about an LLM that you built. I’ve been reading so much LLM output that now I see it everywhere. I wonder if humans will start writing more like LLMs?

brainless 5 hours ago

When we hyper focus on finding something, we find it all the time.

The world of tiny LLMs is so interesting. It is unlocking novel ways to encode information. Why focus on the style of writing instead of the subject matter?

swiftcoder 2 hours ago

> Why focus on the style of writing instead of the subject matter?

Because just as with human writing, a poor writing style obscures the topic where it should illuminate

SubiculumCode 5 hours ago

Anyone can level a charge of AI writing, but it's getting to the point that you must write sloppy to seem human. and never never write well with clear technical prose. It is getting to be just ridiculous McCarthyism.

dakolli 4 hours ago

I don't see it. I sit in a mild state of rage trying to decode llm prose all day, this isn't it.

springtimesun 2 hours ago

I realized the same while reading it, but my impression is that the output has been well edited. The typical Claude density has been pulled out into something more human readable, but with many of the Claud-isms still intact.

Personally I find this a productive way to produce documents. Claude makes the first pass and I edit line by line. It’s a good understanding check. If I can’t reword something into my own voice it means I don’t understand it. Then have Claude loop again to fact check the rewrites and repeat until clear (or I disagree with Claude on all it’s remaining points).

That’s the same edit loop I used without LLMs, but I get speed in drafting up front (and fact/number pulling) and the check passes make the edits better.

woolion an hour ago

Pangram says 50% human. That seems to check out, heavily edited yet the LLM aha sentences and cadence are still definitely there, but not overbearing.

I read some Dostoevsky recently and I found my speech to be substantially altered for a while, so it makes sense that LLM writing would have an influence.

vatsachak 6 hours ago

Very cool. I wonder what's the minimum amount of money you'd need to get it to score well on AIME