Build your own decision model (nishtahir.com)

116 pointsby softwaredoug3 hours ago26 comments

sn0n 8 minutes ago

I saw Jev and without understanding it said to myself, “I can build that!” And brainstormed some weird Alex trebek jeopardy generator I called trebek, a bun ran typescript app that you give it an input, it decides if it’s a category, question or answer then generates what’s missing. Trained a SQLite-vec database on the English language for a few days with a small qwen embedding model to add vec embeds to the db then ripped the cord on the embeddings before I started feeding my llm the proper specs for Jev and now i have a cool jeopardy generator that now doubles as a Jev clone classifier with a custom v1 endpoint for system 0 or whatever it is. Level understanding here, fun experiment though ^>^ and produces useful outputs

ford 30 minutes ago

These are neat - and the source of the many Jev clones we've seen. I think their recent funding round is in part because of their algorithms/data. It remains to be seen if that's a big enough edge to be worth 1 billion+ dollars

sva_ an hour ago

Does someone have examples of interesting stuff that has been built utilizing Jev/decision models? The way this is hyped up surely there must be some good stuff?

nico an hour ago

Not with Jev, but you can use classifiers for a lot of use cases, here’s a few: https://playground.jeffyclassify.com/

nico an hour ago

This is very cool. If you are looking for something similar but more lightweight, that you can run (and train) on CPU, try out Jeffy: https://jeffyclassify.com/

On GitHub: https://github.com/nicobrenner/jeffy

mrkn1 40 minutes ago

Cool project too. If you are looking for a 500MB instead of gigabytes, with evals on Jevbench that you can run fast on CPU check out gutsy.

https://github.com/kouhxp/gutsy

howunfortunate 2 hours ago

As an MLE who has been failing to get anyone interested in classifiers for many years, the hype around Jev makes me scream internally.

Yes, I get that a zero-shot classifier is more convenient than the traditional kind, it's very cool. Kind of. But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now, so again I'm back to my internal screaming.

copperx 2 hours ago

I want to share the rage. Can you expound on what makes you scream?

howunfortunate 2 hours ago

Idk, imagine you worked on Skype's B2B sales team for years and then COVID happens and Zoom blows up.

Is it a better thing? Yeah. Does it affect me in any tangible way? No.

But come on, really people? All you needed was like one tiny bell & whistle to take this from nothing to the hottest thing of all time?

JMKH42 an hour ago

I bet the timing was the key, people in the last year have been furiously building things that use LLMs as an API and as we work on this stuff we have systems with N LLM steps and M of them are frustrating because you want a specific choice picked or list of things ranked and sometimes the llm will just output something else entirely!

So along comes this thing you can graft in that is more reliable, faster, and cheaper for that, and I get it immediately.

A year ago I'd be like "kinda cool but what it for?"

firasd 2 hours ago

I think part of what made Jev catch on is that the API is like an if(...) or switch statement

People are just so used to the chat style APIs that they didn't even consider doing things like sending a bunch of emojis to a chat model and then asking for the optimal one in this context etc. Also chat models are pricier for the same behavior and can also output something random like a refusal

But yeah ironically I think in the initial breakthrough LLM paper on GPT-3 in 2020 some of the multiple choice questions were answered by comparing token probabilities of specific continuations rather than fill in the blank

howunfortunate 2 hours ago

You know, it's a good point.

Maybe I should have spent less time pitching to PMs and more time pitching ground up to devs, who have the right foundation to intuitively understand the usefulness.

jofzar 2 hours ago

> But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now, so again I'm back to my internal screaming.

It's the scale of "perfectly cheap", jev (specifically) is so dirt cheap and fast that you can throw it at things that should not be justifiable in the past and you barely have to do any work other then quick testing.

dominotw an hour ago

> As an MLE

Yea but those models you were building were lame and inaccessible to play with for common devs.

Just because they have the same api doesnt mean you were building the same thing.

BoorishBears an hour ago

Maybe instead of screaming you can take this as a chance to level up your engineering.

Good engineers don't treat approach as A == B or even A like B, when extremely integral parts of their applications differ.

Zero-shot isn't just "more convenient", in a low data regime: it's the only workable solution, and 100x so if your plan involves the acornym "BERT" (because even the largest of those models has the world knowledge of a fart to draw priors from)

Better ergonomics while being faster and cheaper as the existing things really is enough to justify callling what you've done a new thing, in a world of finite resources and time. It's actually making me scream how many people don't get that.

howunfortunate 19 minutes ago

> in a low data regime: it's the only workable solution

Low data regimes no longer exist in the age of LLMs, one can trivially generate a training and eval set and distill a good classifier on any domain within a day.

But I do acknowledge zero-shot is more convenient. Personally I don't think Jev has any moat so I won't bother with their model specifically, but yes I do anticipate using this type of thing more in the future.

win311fwg an hour ago

> But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now

I'm no prompt engineer, but I've found them to be too slow any time I wanted to use them that way. I have never tried Jev, but apparently it is supposed to be fast, so it seems like, according to the marketing, it could become usable where LLMs haven't been.

esafak 44 minutes ago

Jev is classification for normies. Look ma, no training.

laybak 40 minutes ago

on the bright side, maybe this zero-shot classifier wave could be the back bone for more specialized classifiers (with more mindful selection of data and training)

demibabs 2 hours ago

Is simply changing the temperature so that the model appears calibrated over a particular benchmark after the fact “allowed”? Feels p-hacking esque.

JMKH42 an hour ago

if it works it works! as long as the test set is reasonably large and diverse its better than nothing. You could characterize how robust it is by throwing dozens of different types of work at it and see how much the confidence varies

ursaguild 2 hours ago

This is really cool to see. Being able to play the token generation was awesome. Amazing job with breaking down how to think about these models. This made the idea of Jev/decision models really easy to grasp for me. The idea of calibrating the model was helpful. I thought this was a great overview.

bellajbadr 3 hours ago

Is this only about getting fix json output?