We submitted Needle 2 here a few weeks ago, and the feedback in the discussion thread was incredibly valuable, thanks! Thanks to all that feedback, we’ve been able to move quickly to release Needle 3 and I'd love to hear what you think again.
The key features:
1) Automation (tool calls & structured JSON output): Needle still doesn't chat by design, its quite challenging to pack general capacity into such small models, so we focus on tool calls and structured JSON. If no tool you declared fits the request, you get an empty list back (note for when playing with the demo).
2) Intelligence Laddering: Every layer (2 to 20) is a deployable subnetwork, so one set of weights, 25 to 121 million parameters at 2-bit, shipping as 8-29MB binaries. On a Raspberry Pi 5 it decodes at up to 4k tokens/sec and prefills at up to 10k.
3) Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.
4) Performance: On Mobile Actions (phone commands, scored on the exact call) the 20-layer model gets 86.0 through the shipped 2-bit binary; LFM2.5 1.2B is at 82.4, Qwen3.5 0.8B at 76.0, Apple's on-device model at 57.6, all at f16. More results on the link, we do not win everywhere ofc.
5) Multilingual: Needle 3 now supports English, French, Spanish, German, Dutch, Italian, Polish, with more languages coming.
6) Finetuning: You can achieve DeepSeek v4 Flash grade performance on a narrow task with just 4L, stress on "narrow task", we found that production users often prefer tuning before production.
7) Triggers: Grounding is a common challenge for tool call, at least for Needle 2, so we added support case-insensitive regular expressions matched against each request to gate false negatives.
8) Confidence: Every response also carries a calibrated confidence score, the minimum of a judgement on the finished call and its decode probability. Act above your threshold, show the call and ask below it, or escalate to a bigger model.
9) Supported Platforms: macOS, Linux on x86-64, ARM64, ARMv7, RISC-V and MIPS32, Windows x64 and ARM, Android, iOS, watchOS, tvOS, the browser as WebAssembly, and a WASI component.
Thanks for reading and as always, thoughts appreciated!
IanCal 16 hours ago
Wondered if it'd turn on the lights in the bathroom with these:
"I need a wee" -> tries to play music because "wee" is a genre
"I need a wee wee" -> starts the vaccuum in the bathroom
"I'm going to the toilet" -> says it'll turn on the toilet, and I'm not totally sure what that entails.
"I'm going to the toilet and can't see" -> reasons that lights should be on in the bathroom, then chooses again to turn on the toilet.
"I'm going to the toilet and can't see where I'm going" -> reasoning is "'going to the toilet' -> control_device with device 'coffee maker' (toilet implies coffee maker)"
"I'm going to the toilet and can't see where I'm going because it is too dark" -> "'dark' -> direction 'dark'; adjust_lights with brightness 100 for darker light"" and chooses to turn the lights in the living room to "dark" which fails.
At this point the vacuum is in a dark bathroom, the living room is 100% brightness and playing "wee". At least there's coffee.
HenryNdubuaku 16 hours ago
thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!
IanCal 14 hours ago
What kinds of things do you expect to work?
Edit - I’m struggling to get anything useful. Reasoning is often utter nonsense and the actions are very often very wrong. To the point of seemingly needing very precise sentences to work at which point you may as well do regexes. Very simple things like clean one room then another with the vac fails.
HenryNdubuaku 13 hours ago
Thanks for the feedback! Implications and relations are hard for the model to understand (things like go to the living room, then the kitchen, and back), so yes the cleanest use cases involve direct language. Reasoning isn't true reasoning in the way general LLMs do it, it is more like grounding for the model that it generates itself. This can often become nonsensical specifically when the model gets things wrong, providing signal to the confidence.
rohansood15 an hour ago
Can you share an actual example of where it works please?
ash_091 13 hours ago
Pretty much matches my experience.
> 'sleepy time' means sleeping → start_vacuum with room 'bedroom' to start cleaning
The "DeepSeek 4 Flash grade" claim seems far fetched.
electroglyph 8 hours ago
those are all expecting far too much for models this size
p1necone 4 hours ago
Given the title of the post says 'can match deepseek v4 flash' I think it's fair to call out these sort of dumb mistakes.
mentalgear 3 hours ago
Maybe with fine tuning?
gs17 16 hours ago
"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't. "it's too cold" actually made it turn the thermostat down ("it's cold" made it... turn the lights down?)! Although the confidence on the bad responses was pretty low, so it might be worth adding a threshold to the demo.
Or maybe it just has a weird thermostat down bias? "make it hot" also had it turn it down (specifically it went from 20->18, or at least tried to, the UI still showed 20), with high confidence. Also might have a bit of a Celsius vs Fahrenheit confusion. Neat concept, but I might not want to let it control the oven at the moment.
The laptop demo worked better until I tried to open the mail app. "Check mail" kept opening the browser with an error, and "check email" makes a note with the text "email", "open email" goes to "https://api.email.com/v1/email" in my real browser, but "open mail" does work.
And I presume the "reasoning" isn't very trustworthy? In the car I got "'turn it up' means lower volume -> set_volume with lower value." For the house, reasoning would correctly say that I wanted the alarm off, but it didn't actually do it.
HenryNdubuaku 15 hours ago
Hey, thanks a lot for this feedback, very useful and actionable for us! Quite a few of these came down to our tool definitions in the playground as well as out triggers. We updated them just now and these should be more reliable. Really this goes to show that needle shines through after putting in the work to make the tool list around it good for your use case.
As for the reasoning, yes its main function is really to provide more words/keywords that the model can latch onto when generating the tool call response, since this is a SAN model it needs more grounding in existing context.
IanCal 14 hours ago
I can’t help but wonder how well more traditional approaches would do with this. Something like a map of statements to actions, with fuzzy search - then remove what used to be the labour intensive part of this by handing it to a decent llm to generate the sentences.
HenryNdubuaku 14 hours ago
That's a really good point and I think it's not yet clear how well, say, 8-30MB worth of regexs with accompanying algorithmic structure would do on these tasks. I would imagine they do quite well on a well defined task, but it would be much harder to then adapt this set to a new domain. A big part of Needle's promise is how easy it is to finetune. Ultimately I think the two approaches can be more complimentary to each other, rather than choosing only one (see triggers!).
potatoman22 12 hours ago
I think a good "traditional" approach would look like a BM25 algorithm over an index of trigger phrases for each category, sitting behind a majority-vote classifier. The "fine tuning" would be done by reindexing the data, generating different/new phrases, and tuning the classification threshold.
HenryNdubuaku 12 hours ago
I think we might look into creating a baseline like this for our future models
raybb 16 hours ago
I have an idea for a use case for this, and I'm wondering if you think it makes sense or if you have any thoughts on the approach.
I'm a big fan of OpenStreetMap, and I enjoy editing it from my computer. From my phone, I find it quite tedious trying to make sure I type in the phone number exactly correctly and double-check it, or find and select the right field from the large list of fields available in Upredor.
Generally, how it works is I see a restaurant, and there's a sign. I know that it says, "Cash only. Here's the phone number. Here's the opening hours." What would be really cool is if I could just speak to the phone and say, "Hey, here's the information about this place." It would automatically use your location to detect what places are nearby and maybe even detect which place you're talking about, and then tell you, "Okay, here are the changes I think you're proposing to make, or these things you stated are the ones that would create a diff." This would be limited to just perhaps the 20 most common keys in some predefined set of values for most of them. Like cuisine=x should just match to the most common not make up new ones.
Of course, this is something a large language model could do, but having it run on device would be a lot nicer and cheaper.
HenryNdubuaku 16 hours ago
Makes a lot of sense! declare one record with the ~20 keys as fields, cuisine and friends as enums with the common values, and the grammar can't produce a value outside the set; fields with no evidence come back empty, so the output is exactly the diff.
For "which place", query nearby POIs from location in the app and pass the candidate names as an enum field, so Needle picks rather than guesses.
Two caveats: it's text-in, so you need on-device STT first, and opening_hours syntax is the risky bit, so either put the format in the description or capture the raw hours and normalise in code.
hirako2000 16 hours ago
My thought, the growing number of dubious claims that a tiny model beats LLMs will make any useful innovation be overlooked.
What's more important than the resource requirements is to highlight what the model simply cannot even attempt to do that general LLMs do decently well.
In other words, tell me the anti use case clearly so that I don't have to find out myself.
HenryNdubuaku 16 hours ago
Strong point! Needle is a task-specific model and bullet 6 stressed that it is only trained to be good on a set of narrow tasks, but I guess it could be clearer?
owebmaster 16 hours ago
It would be clearer if you didn't use AI to reply.
janalsncm 16 hours ago
An LLM is a Swiss Army knife. This is a corkscrew.
All of the other tasks a general-purpose LLM can do (write me a poem about pizza, rewrite this code in rust, tell me about the causes of the war of the roses) are unsupported.
The only use case this supports is converting unstructured text into structured json calls, and doing that quickly in a low memory environment.
Scaevolus 12 hours ago
This is a solid improvement over Needle 2, which I tried using for a tool-calling interface to a Runescape database site. Unfortunately it's still not quite capable enough for my target compared to FunctionGemma.
Thanks for testing Needle out! I'd be very interested in hearing more about the finetuning setup to see how we can make both the library's finetuning setup and the model better.
neilellis 14 hours ago
I tried this today for labelling - and for that task it was very bad MNLI was better - so you are going to need to match the use case for this pretty exactly. (at 29MB params one would expect that!) I'm obviously not saying labelling is a good use case :-) just adding a data point.
Jev has put the cat amongst the pigeons so suddenly everyone is looking at classifiers and encoder only models again.
My ideal model would be a general purpose LLM API that can answer classification questions and as it does so distils to an encoder only model so that the more classifications I do the cheaper it gets (i.e. the more it offloads to the classifier). If anyone ever wants to do this as a service do let me know, because it's just another piece of code to manage in each new project that needs classification.
Also a model that could do this internally would be nice :-)
HenryNdubuaku 14 hours ago
Hey! Yeah I think for labelling the model would need to have much better world knowledge than its current size allows. Jev really is a very good model, I think it has a very strong place in the upcoming tech stacks. Really good suggestion to make a continuously distilled model, we are going to have to look into that one :)
neilellis 11 hours ago
Good luck with this model/product, in the excitement of LLMs people seem to forget applicability. I very much like to see innovation in this space, so well done!
Retro_Dev 12 hours ago
A very cool project, but of course not perfect. I'd rather have 30 megabytes of phrases mapped to the perfect and correct control changes in a home, rather than a heuristic built around 30 megabytes. I tried to "warm the house" (increase the temperature of the thermostat), but the model actually turned the lights to a "warm brightness" - reasoninig being `"'warm the house' -> set_lights to warm brightness. No specific room given, so use default 'living room' as default."`
HenryNdubuaku 12 hours ago
Thank you fr this. "warm the house" now goes to the thermostat. It's fair that a more deterministic system with just action phrases would be easier to debug/interpret, but I think there is room for both a model that is trained to understand meaning as well as deterministic logic aiding it. To this end, we just started exploring the idea of triggers, and are working towards expanding this even more.
janalsncm 16 hours ago
Hey, I’m really happy that someone is building this. I tried doing something similar a couple of months ago and came to the conclusion that the dataset was at least as important as the modeling itself. Building a good dataset is nowhere near as flashy as building a novel model architecture, but it really is critical.
For instance, you want to be able to handle any smart home commands people could issue, right? What are all of the smart home devices? What are all of the ways people might want to issue commands? Also, for things like Spotify, it’s not going to know what “The Beatles” are or “Led Zeppelin”. Artist and song names themselves are easily just as hard as all of the smart home devices combined.
The simple attention network stuff is cool, it makes sense to drop the MLP when it dominates the param count. But you’ll definitely lose some “world knowledge”. That’s probably ok though.
HenryNdubuaku 16 hours ago
100%, data was honestly most of the work, Needle 3 is trained on 360B tokens of structured data and we spend way more time on the generation pipeline than on the model. On Led Zeppelin, Needle doesn't actually need to know it, arguments are copied from the request so it just lifts the name into the artist field. The knowledge went into the engram btw, 70M of the 121M params are n-gram tables, so it can tell artist vs song without an MLP. Also yes, "play their second album" won't work, that needs the world knowledge it doesn't have.
jamiesonbecker 10 hours ago
This is incredibly cool. I tried telling it to both turn up the temperature in the home and turn down the lights in the LR and got a beautiful JSON doc.
Esp paired with a small Whisper or Parakeet voice model, this enables some amazing real-world use cases in lower-power situations (car, marine, home, PLC, industrial automation). Combining with a solar/wind combo for your home could be incredibly interesting.
HenryNdubuaku 9 hours ago
thanks!, let us know if you ever build it out :)
digdugdirk 8 hours ago
I've seen previous needle releases running on esp32s - any idea how well this new one would run on something like the newer esp32-p4?
rglover 14 hours ago
This could be really handy for triggering admin functionality. Would be kind of nice to just quickly be able to say "Reboot <server>" to a chat instead of digging through a GUI (especially when you're remote/in the field).
HenryNdubuaku 14 hours ago
Oh yeah really good use case! Definitely something to finetune the model for so that it gains better task-specific reliability, because rebooting the wrong server could easily be catastrophic.
rglover 14 hours ago
Is there a way to get in touch and chat about use cases? I have some low-stakes stuff I could test this against without causing a meltdown.
HenryNdubuaku 13 hours ago
I think that would be very useful for us! The best way to reach us is through the [email protected] email
Thank you!
asaddhamani 13 hours ago
What is 8 dash 29 MB? And the copy on the landing page is clearly AI generated with the “each layer a model of its own” stuff, makes little sense. The more I see AI generated copy the less it makes sense.
HenryNdubuaku 12 hours ago
The model can be sliced and perform the inference using a subset of its layers. The first 4 layers alone are 8 MB, all 20 are 29 MB. Fair point on the copy, tightened it.
Tsarp 16 hours ago
Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it makes sense to run something like a 8-27B on local network and get the work done.
Are there perhaps some industrial or agri use cases?
HenryNdubuaku 16 hours ago
Fair, we gotta do a better job at explaining this properly!
So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.
yorwba 15 hours ago
A list of hardware platforms doesn't make a use case. Do you have an active deployment of Needle that is noticeably useful, and if so, what do you have it do?
stymaar 13 hours ago
At my current company we're evaluating small models embedded directly in the web app to provide a natural language interface to the app without spending money on inference (and ideally avoid a ChatChipotle situation where people end up having free token going through our interface).
And needle is one of the most promising model due to its original architecture (but we still need to finish building the actual eval dataset before making out final call).
HenryNdubuaku 12 hours ago
Thanks for considering needle. Keep in mind that you can also fine-tune the model to fit your use case more. I think this illustrates the intended deployment pretty well, where both computational resources and compute credits can both be issues for deployment.
stymaar 2 hours ago
We definitely intend to try fine-tuning, don't worry we're not going to dismiss needle just because the base model's performance is too low ;).
int_19h 3 hours ago
You can embed much bigger models into web apps with wasm and WebGL or WebGPU. I have a web app running a 0.6B embedding model client-side.
stymaar 2 hours ago
WebGPU is a non-starter for production usage since the support is too limited (No Firefox support, no Linux Support, no Apple x86 support, no MacOS <26).
But yes we are also considering bigger models, though we'll pick the smallest model of sufficient quality because not having to download a 600MB bag of weight is a feature in itself.
ash_091 13 hours ago
So it will run on those devices, but when/why would you do that?
All of the demo setups (smart home, robot vacuum, watch, etc) could easily have access to a bigger model running on a more capable device either locally or via the internet.
KennyBlanken 4 hours ago
You've explained WHAT it could run ON. You haven't shown any examples of its intended use, from input to result. I have no idea what is expected of me to be entering into that home automation example, and as everyone has handily demonstrated, it's dogshit at what people actually try to feed it so clearly im not the only one.
mihau 17 hours ago
None of the queries I asked worked:
- "more light"
- "less light"
- "both doors should be locked"
- "if blinds are open, open back door"
HenryNdubuaku 16 hours ago
thanks for this actually, so the demo is a preset, you can edit tools descriptions and add available tools the way you want, else Needle heavily guards against false negatives, users asked for this. For this, we will update presets on our end.
HenryNdubuaku 16 hours ago
presets updated for you now :)
mentalgear 3 hours ago
Feels like Jev, no?
Natashash23 13 hours ago
Really interesting project. The intelligence laddering and on-device tool calling are especially cool. Nice work getting this running across so many platforms!
HenryNdubuaku 12 hours ago
Thank you!
xmcp123 14 hours ago
“The cat puked near the refrigerator”
Made the robot vacuum clean the living room. Might be good to give it an idea of where items likely are?
Very cool though. I see a lot of potential.
HenryNdubuaku 14 hours ago
Thanks! Certainly giving the model more context on the task it needs to perform would help it. This was actually a part of training that we improved going from Needle 2 to Needle 3
xmcp123 7 hours ago
Just wanted to say it’s nice to see some people working on an AI that just objectively can make people’s lives better.
It’s not replacing anyone, it’s not going to destroy our energy infrastructure because that was the only way to turn off a toaster that it wanted to turn off.
Nice work.
sroussey 10 hours ago
Would love to see this implemented with @huggingface/kernels for shader compilation for Webgpu.
I think it will be the basis for a rewrite of transformers.js v5, but no need for you to wait as you would likely want direct access. It is also way better than loading WASM, and faster to boot!
razster 12 hours ago
"Illuminate (roomname), de-illuminate (roomname)" works well.
Harden perimeter, locks doors and sets alarm.
Neat stuff.
HenryNdubuaku 12 hours ago
Thanks!
eriwang915 5 hours ago
Tool-call + structured JSON at 8–29MB is the right bet. Returning an empty list when nothing fits beats inventing a fake tool.
HenryNdubuaku 4 hours ago
thanks, we improved on False Negatives this time :)
dizzard 14 hours ago
Anyone know if the model architecture overlaps with Jev, or is it just coincidence these are releasing at similar times?
HenryNdubuaku 14 hours ago
As far as I know Jev's architecture isn't public (though I might be mistaken!), but it is a coincidence :)
Needle 3 has been in the making since Needle 2 launched early august, but we are very excited that Jev is bringing more attention to the problem we are trying to solve.
monster_truck 15 hours ago
I told the "car" to close the garage door and turn the car on and it didn't immediately refuse
HenryNdubuaku 15 hours ago
haha i think that's a good demonstration of how external guardrails could help ground tiny models like these to prevent issues from coming up. I wouldn't trust needle to be my autopilot either (:
monster_truck 14 hours ago
I don't think you're taking this seriously enough if you think external guardrails are sufficient.
HenryNdubuaku 14 hours ago
well certainly the environment on the website cannot be a full product, and it isn't claiming to be that. The model, while capable in many dimensions, is also limited by its size. The website is meant to show both the capabilities and the limitations! A real deployment would absolutely need external guardrails, more thoroughly thought out tool sets with better task-specific triggers, perhaps also task-specific finetuning for better confidence grounding. And in my view that's the point of small open models! You can take it and run with it as far as you want.
hypfer 15 hours ago
"Turn the kitchen to 230°C" was executed with "confidence": 0.9015
HenryNdubuaku 15 hours ago
lol well i guess you can turn the whole kitchen into an oven with needle :)
But for real usecases you are able to set explicit minimum and maximum values on the output range of numeric arguments, so that you can avoid situations like these. In this case it was hard for us to do that while keeping a broadly appealing demo since celsius and fahrenheit have different "reasonable" output ranges.
neilellis 14 hours ago
Also FYI doesn't run inference on Apple GPU (only for training)
HenryNdubuaku 14 hours ago
Hey there, yep we found that on Apple devices specifically running on CPU is fast enough that Metal support is not needed. Thanks for flagging this though, and if usecases that would benefit from Metal support come up we will be adding it to the binaries.
sourcecodeplz 15 hours ago
from my limited tests, it can work with up to 10 tools/definitions.
over that and it gets confused
HenryNdubuaku 15 hours ago
Hi, thanks for the feedback! And yes absolutely less tools and better tool descriptions make a huge difference for this model.
