Jev in 25 Lines of Python (nobodywho.ai)

Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.

To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.

I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.

0: https://arxiv.org/pdf/1706.04599

https://til.simonwillison.net/llms/llama-cpp-python-grammars

https://sgnt.ai/p/jev/

I got this technique to work extremely reliably last year. However there were a bunch of caveats: 1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality. 2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure. 3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't. 4) You must be exceedingly careful with tokenization.

But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.

Prompt part: "What is better, toast or bread?"

Incomplete answer part: "The answer to this question is "

and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.

There, I fixed your problem.

Payroll sends you an email with a link to a Youtube video that plays a song.

Options after body:

    Average probabilities:
    Rickroll   0.5158 ( 51 wins)
    Phishing   0.4561 ( 47 wins)
    Spam       0.0281 (  2 wins)
    Joke       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

    Average probabilities:
    Rickroll   0.9293 ( 94 wins)
    Joke       0.0549 (  5 wins)
    Phishing   0.0140 (  1 wins)
    Spam       0.0018 (  0 wins)
    Legitimate 0.0000 (  0 wins)

EDIT

Gemma4-12B-it-NVFP4 seems way less sensitive to option/body ordering:

Options after body:

    Average probabilities:
    Rickroll   0.9867 ( 99 wins)
    Phishing   0.0133 (  1 wins)
    Joke       0.0000 (  0 wins)
    Spam       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

    Average probabilities:
    Rickroll   0.9401 ( 93 wins)
    Phishing   0.0336 (  3 wins)
    Spam       0.0250 (  4 wins)
    Joke       0.0010 (  0 wins)
    Legitimate 0.0002 (  0 wins)

>calls an api

ok

you can swith to a better model for lower error rate.

Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.

Non deterministic systems have furthered the "brain rot" in our industry.

Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.

The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.

And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.

"Trust me Bro"...

it is the latency that makes it significant

"System One" and "System Two" were coined in some pop science book...so back to its usage being a marketing ploy.

Either way it's an analogy that's bound to be loose as Kahneman's modes are about humans.

Huh? I guess that depends on the exact definition of "classification", but I think the bulk of basic classification tasks we make every day to make sense of our surroundings, such as object recognition is definitely done using system 1. So is higher-level "stereotyping" or anything you could described with "I know it when I see it".

Because those responses can be incorrect or even harmful, you would sometimes make use of system 2 to correct them - but that doesn't change that the initial response is from system 1.

I'd argue that most human classification is pre-conscious / System One. You see a table, you recognize it as a table without asking yourself "is this a table?"

I guess their marketing implies that it moves classification into system one response time.

I think you answered your own question. Executives are going to ask two questions, 1) how is this different/why does it matter and 2) how will i use it to make money?

Leya came to market more than a year before Jev, and failed because nobody understood how to use it, and he was unable to market it properly. Jev used this strategy and did not fail.

So, fast in the LLM space and comparable with Jev.

> But their example is classification but that would also be possible and faster with a classic BERT model.

With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.

> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?

Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.

Not particularly. There is still the problem of hallucinations and varying results across runs.

That's more of what type-safety means for their team. Every run gives the same results. It's type-safe

As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size

If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.

But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.

- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased

- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking

- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.

If you are using structured output, I think those 3 together is a really big deal.

>But their example is classification but that would also be possible and faster with a classic BERT model.

I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.

While technically correct, it's not the same thing

Ultimately Jev claims to have a data advantage which is likely where the future lies. They'll have a unique edge in improving general purpose classification / decisioning.