Kolibri: A Sovereign Open-Weight Model (aleph-alpha.com)

His point about regulation and innovation is great and I wish more people thought like that.

One of humanity’s biggest problems here is we don’t know how to do moderation.

We have two modes. One is a brick taped to the accelerator and damn all consequences, driven by national pride or corporate greed or egos. The other is a brick taped to the brake driven by histrionic doomers and anti-everything pessimists.

The extremes are loud and fit in a tweet. Nuance is quiet and contemplative and usually requires an essay or a book. It’s also dynamic. Nuanced positions evolve over time as new things are learned. Extremes tend to be fixed and rigid. All this, I think, gives them higher memetic fitness in the discourse.

I don’t think this is new. Look at nuclear power, a largely pre-Internet example. You had pro nukes who minimized and hand waved away any risk and anti nukes that wanted it utterly outlawed. Nobody said “hey this is a great zero carbon source of energy but we really need to think it through carefully and manage it well.” Or if they did they were drowned out by the loud screaming extremes.

In my opinion not being open about which data is ingested and trained on, and trying to make that a repeatable thing for a third party, is not worth being called "open". Glad they did that.

This alone makes it much more valuable than many high-profile releases despite not quite performing at the same level.

Hopefully this becomes the new standard.

It’s seemed crazy to me that anyone thought these could stay closed or even SHOULD be closed source.

Yeah the pdf alone is awesome as a learning tool.

I worked on Kolibri, in particular pre-training data and mid-training. We strive to be as open as possible. Glad you like it.

The paper mentioned is here: https://tej.as/blog/aleph-alpha-kolibri.

(This comment was originally posted to https://news.ycombinator.com/item?id=49943034, but we're merging the threads.)

Such a crazy change from the times of Luminous, when they published a three pager with a claim that the model is similar good as „gpt 3“ (which??) with some graphs without y axis.

Bravo team!

I wish universities would take it upon themselves to curate the training sets for these models.

Gemini gives me a lot of hard-no responses, and I think to myself, "well can you find out? what am i paying you for" usually

Gemini: Here's a list of links to check

Just like Merl.

https://minecraft.wiki/w/Minecraft_Support_Virtual_Agent#Sys...

(for context, the Minecraft Support chatbot, aka Merl, had a meme because she kept saying "I don't know" to questions like "How to craft a diamond pickaxe".

I'm not super impressed, at least in English. I asked:

  What is the canonical interpretation of the song ‘Glass Flowers’ by Armand Uso?

(A made-up song and name) And received:

  "Glass Flowers" by Armand Uso, a track from the 1978 album The Art of Falling in Love, is generally understood as a melodic reflection on the fragility and impermanence of love. [...]

I saw similar responses for other questions. Qwen3.8-27B correctly refused without web search.

Not impressed. I asked it how to run itself (giving it the Huggingface link) on limited RAM i.e. less than stated as needed and on llama.cpp and true to what we read about "it will tell you when it doesn't know" that's almost all I got: It doesn't know, it told me I should go click on tabs in the Huggingface interface for more information. This was with extended thinking on.

No, I'm not gonna do that, I asked you to do that Mr Kolibri.

Also feedback on that interface: It's very annoying while answering. It almost immediately shows a list of sources, which on my screen fill up all the space and then when it starts answering it keeps those in view but also scrolls down the tiny part of actual text its outputting but I can't scroll up to start reading from the top, coz it keeps scrolling. I have to wait until it's completely done generating its output.

Friendly note: your website's font at its current size is pretty bad on a non-retina display before zooming in.

It's been surprisingly good at the things I threw at it (history, culture and conversing, web search)!

- Is it possible to train only on math and logic materials and expect the model's response in math questions to be superior to general models with the same training/inference compute hardware?

- Are there non-LLM approaches to the above task with the goal of achieving a non-hallucinatory agent?

Given that "sovereign" these days implicitly means independence from USA and China, and Canada being spiritually in the same boat as the EU, it's pretty spot on?

The costs of "keeping up" aren't really growing, on the contrary.

Hey there, I worked on Kolibri pre-training data. Agree with the point that non-US/non-Chinese labs need to pool effort. The merger with Cohere is public, nothing to hide and I'm also excited about it personally.

I was also quite surprised to see that. Considering the effort Aleph Alpha put into to their new model, it seems like the Qwen team needs access to vast amounts of german data o.0 I'm really glad for these efforts for open models from within Europe.

Well, it's still independent from the USA and China.

The main problems with big corp AI are due to control of access in the first place and control of what they output.

When you make your industry reliant on such choke points, you render yourself the opposite of "sovereign" for sure. Having multiple independent suppliers at least mediates that.

Yeah I think a model has to be actually good to claim sovereignty, as in competitive enough that people want to use it. Chinese and US LLMs are the only ones in these categories right now. Mistral and Cohere have the same problem, yes they are made in different countries but they are not competitive. They (France and Canada in this case) would be better off just downloading Chinese LLMs, even if they get cut off they still have the weights.

I’m most familiar with Canada, where sovereign is usually just an excuse to overpay someone connected for an inferior product with no strategic value.

Disclaimer: I am part of the team that trained Kolibri, opinions are mine.

Qwen models have solid performance on benchmarks and we're transparent about this in the report. We're just happy to share a European alternative in the small model space, where I don't think we can afford to be fully dependent on China. We also put a lot of effort into German language quality things that don't show up in evals at all (style, Grammar, German reasoning, etc).

Sovereignty is imho mostly about choice and control over your data. Cohere is no different on that front and personally I'm quite excited about what we will build together post-merger.

The issue with 'sovereign' is not about sneaky things a foreign entity might put in there, it's about control of the stack.

Since no individual buyer cares that much about geostrategic issues, it's almost impossible to get some random European company to think about buying anything other than 'whatever US or China' are making.

There has to be very concerted push to make a difference.

Legal mandates around sovereignty (be very careful here) could make a difference.

But there will also be market reaction: US/China companies will bend on some level and provide things like 100% EU hosted and even comply with some data source stuff.

The only real path is to be more competitive as a continent.

That is a very good idea and there is a market for that, especially if a company manages to source the hardware and then install the stack on a german or EU customer site

It depends on the use case, but I believe SLMs can do great with smaller tasks. People got so attached to 'general purpose' that they forgot software can be designed for specific, smaller use cases.

Can't you infer the comparisons you would like from baseline results provided? There are better results in the sibling post on HN: https://aleph-alpha.com/en/blog/kolibri-has-landed-a-soverei...

That is what stood out to me as well. Qwen 3.8 Flash can run on very limited hardware as well and it is at least as good as Sonnet 5 in benchmarks like DeepSWE. With its n-gram design you can get flash next running on very limited GPU resources, as little as 16GB of VRAM.

People follow the latest frontier lab models with great attention and migrate to the next big model on their subscriptions. Meanwhile these local models have quietly gotten REALLY good. It is not even an exaggeration. It has happened in the last couple of months.

"Local model you can run at 40 t/s on a gaming machine that is better than Opus 4.6" is way less exciting than "OpenAI IS DOING CRIME!!! OpenAI SOLVED NAVIER STOKES. DARIO SAYS GLM 5.3 BAD! SLOW DOWN THE FRONTIER!".

(edit: also... totally ignore that 27B dense column over there where Qwen 3.8 27B beats Kolibri on nearly every single benchmark. Why would I choose to run this model?)

> it spends way too many tokens on overthinking stuff

Yeah. It’s a German model.

Did you try lower reasoning modes as well?

It’s an incentive problem. If “sovereign” becomes your claimed value proposition, you can claim success even if the models not competitive, so nobody is pushed sufficiently hard to actually make it good.

Sovereign works when talking about building a commodity supply or something, not in literally the world’s most competitive and fast moving field.

Those seeking sovereign capability would be better off aiming to be best at something, even something much narrower than an all round LLM. Or just fast following and making something that matches leading performance, which is close to what the Chinese labs do currently.

It doesn't help that Qwen3.8 27B is an excellent model.

Hi there! I'm Tom. I worked on Kolibri at Aleph Alpha :)

I think it's not simple to directly compare a 27B dense model with an MoE model like ours. As we know, dense models need all params active for every token. Whereas, MoE models (especially sparse ones like Kolibri) fewer active parameters and correspondingly less compute per token.

Among the MoE models we compared against in our tech report and model card, though, Kolibri performs very well in our evaluation, including against models with 12B active parameters. It also best model in the group we tested within that range of active params.

So, I think it's fairer to see this as a trade-off. Kolibri needs less compute per token but more memory, while Qwen3.8 27B needs far less memory and more compute per token. In the report, both are actually on the quality-vs-serving-cost Pareto frontier among the models we evaluated, just at different points.

FP8 weights are actually the default and what we trained for: http://huggingface.co/Aleph-Alpha/Kolibri-1

We have a separate repo for BF16: Kolibri-1-BF16. Will still be tough to put it on a Mac though :(

Disclaimer: I'm part of the team that trained Kolirbi

Have you read the non-English output of those mutlilingual models? They say they put extra effort in to improve nuances and tone in this one. That alone is worth having in many settings.

And related thread on the actual model post: https://news.ycombinator.com/item?id=49942706

I'm looking for this too

If you mean hosting and serving all the other elements of the agent stack, then, yes. We handle sovereign data, so part of our whole value proposition is that every part of our pipeline is hosted in Canada.

This is a blog post by some "random" dude right? The company webpage appears to be German (at least when accessed from Germany: https://aleph-alpha.com/)

This doesn't seem to be true. There's a clear legal path via the first-sale doctrine to train models on copyrighted works. It's been years now, and publishers still don't seem to be offering anything for training (e.g. bulk licenses solely for training use), but adversarial interoperability via cutting up books and scanning them remains perfectly legal.

There's also the ability to distill other models, which is also not illegal (though I'm sure they like to come after whomever for TOS violations, but thats a civil matter).

And, of course, the obligatory copying-isn't-theft observation. A recent supreme court judgment put it well.

> Since the statutorily defined property rights of a copyright holder have a character distinct from the possessory interest of the owner of simple “goods, wares, [or] merchandise,” interference with copyright does not easily equate with theft, conversion, or fraud. The infringer of a copyright does not assume physical control over the copyright, nor wholly deprive its owner of its use. Infringement implicates a more complex set of property interests than does run-of-the-mill theft, conversion, or fraud.

Folks are pretty smart here, I think we can handle these nuances, even if we don't agree about whether they are good.

Edit: reading through the full text of their post, it looks like they are using common crawl, which is likely just as much of a copyright infringement as Anna's Archive -- it's not like published works have a unique claim to copyright. I think this strengthens your point, though: I was expecting to see scans as training data, but it doesn't appear to be the case.

Is it noble? The entire notion that training data can be "stolen" at all is quite silly. If I "steal" content that someone created to use for training, what am I actually stealing? They didn't lose anything. They still have everything they had before. What was "stolen" was "unrealized profit", or put another way: money that wasn't theirs, that they had no entitlement to. The only actual crime that is committed is "unauthorized copying", not stealing. Support and enforcement of copyright feels wildly authoritarian. It's hard to see it as noble.

Have you tried the model itself and seen if it's "even slightly competitive" or not, and have specific complaints about it? Otherwise it feels like you're complaining about something that is easy to test but rather than taking the time to actually figuring that out first, you're arguing about some general and theoretical thing which the submission (may) directly disprove.

This model isn't terrible, at least on the benchmarks. It's 78B A3B and performs about like Qwen3.6 35B A3B. You can probably run it comfortably in 96B of RAM with a decent quant that doesn't lose too much.

Unfortunately, Qwen3.6 35B A3B isn't really a useful coding model. You'd probably want Qwen3.8 27B at a minimum, which requires at least 32GB of VRAM (not system RAM) to run semi-comfortably.

So this isn't going to be a competitive model for hobbyists, and you'd have to be a bit desperate to use it for coding. But if you work in a regulated industry and don't mind paying for a bit of extra hardware, it isn't catastrophically bad, either. Probably would work fine for information extraction or as a "classifier" like Jev. (Almost any GGUF model can be turned into a classifier using llama-server. See pi.dev codemode for sample code.)

So they're not a real contender yet, but they look like they're probably at least minimally credible.

hasn't IP law passed the statute of limitations? As in most models are probably trained on output of other models, as creating enough data otherwise is not feasible. Additionally, they are trained on github repos made since the AI boom, which were generated by models with IP issues (who knows what and how).

Thus training on 'clean' data is like trying to unscramble an egg.

There is no word for copyright in Mandarin :)

Not really. The upside of competitive newer models is all in the proprietary data they are trained on. This is why data labeling, RL environments et al have been such a big industry, OpenAI and Antrhopic are paying literally billions to get the data they need. Do people think the ability to do research level math or advanced cybersec comes from just training on more public data?

A real sovereign effort could invest heavily in this, whatever people accuse China of “stealing” I’m sure they are also generating tons of their own data and are probably the primary sovereign doing so outside the US labs.

Disclaimer: I am part of the team that trained Kolibri, opinions are mine.

You're right that especially big models benefit from training on copyrighted material in terms of world knowledge (especially from books). However, in the small model space imho agentic capabilities where the model looks up knowledge on the fly are much more important. That's what we focused on quite a bit during training. Personally, I also don't think stealing stuff is okay.

Why should we concede this "stealing" framing? If I want to put texts into my computer program, why should that count as copyright violation? I think that idea is as ridiculous as saying that reading a document is a copyright violation.

Regulate large cloud services and proprietary software - yes! But not on the basis of "Intellectual Property".

The "legal" issues here are very very complex and we should not passively wait for or accept corrupt court rulings, international trade agreements, proposed laws, or worst of all propaganda that pushes a parochial and craven view on this.

Thanks! What matters to us right now is not so much our current position, but the velocity with which we’re moving. Kolibri 1 is the first measurement of our position, there’ll be more. We want to build this in Europe and this is only the start. The report also shows our focus on building out proper tooling with our Model Factory.