OpenAI is well positioned to fast-follow Jev (arcturus-labs.com)

Every major AI shop has a ton of in-house classifiers already, big, small, generalist, specialized. Some are used in inference pipelines (e.g. safeguards), some are used in data preparation, training, analysis and investigation, research, various one-off and intermediate tasks etc. Offering them on a public API doesn't always make business sense. I don't see much substance to this buzz, looks like people that are new to all this are discovering that classifiers exist, they are more efficient at classification, and many tasks commonly done with generative models are classification in disguise. Which is not bad at all, a fresh look at their use is great to have.

This article is extraordinarily hard to read. It’s tummelvisioned on OpenAI and things like tool calling which are only relevant to the extent that llms have been tuned to make relative choices, but this applies to all LLMs. Also, some really outdated references. LLM written, perhaps?

Also, moat discussion is the lowest form of discussion. I don’t care if jev has a moat. Did it get the interface right? What other past ideas have we overlooked that if given some love, could kick the door down like jev did?

Really silly stuff.. people wanting to talk about moats when there’s no castle. Moat talk merely projects the illusion of being engaged but, much more often than not, it’s hollow engagement.

I find it unlikely. OpenAI is all in training models with reasoning with RL, and Jev-like models are the total opposite. They are made to not reason at all to be fast. If you want to add reasoning on top, you might as well use a conventional LLM because you lose the price and speed benefits when you output auto-regressive tokens. I don't think OpenAI will even bother with this.

> My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.

Not proof that this is the case with Jev though. It might use non causal text encoder for the state, which could make sense given that it's very good for its price.

There is one benefit that Jev has, that it is not OpenAI and thus it's probably less likely to steal your own work.

Why would OAI need to follow Jev? I really think this is paid by Jev. Jev itself won't have lunch money in a shortwhile because there are literally 10s of free alternatives available which can be run locally on basic consumer hardware. Terrible utility aside, there is no sensible business proposition in Jev.

For context on what "Jev" is: https://news.ycombinator.com/item?id=49717558

If OpenAI releases something similar to what Jev does, then that would be like admitting defeat. Their whole spin is AGI and world ending danger. Why would somebody with an AGI at home make something like Jev which is intended to be a part of some SW the AGI is going to replace anyway.

    > so why invest into a more complex solutions

I'm not exactly following through with the claim, can someone explain how the built-in classification would not necessitate more tokens used, or be much different from turning on reasoning? Not that I don't see the difference, I just doing see how OpenAI would do it well.

Rather than focusing on OpenAI in particular, let's just say that there are many smart people at other AI labs and if it seems like it will be popular, this technique will probably be copied. What would prevent them from adding another API?

Hopefully there will be some decent benchmarks and gateways for switching between providers easily.

comments are pretty weird here, there's no real moat to what jev is doing, it is certain that frontier labs are going to release their own jev and there are even open source alternatives (although nowhere near as accurate as jev).

so maybe typesafe's real plan is to front run and releasing their own new models for some time until they can get acquired which seems to be the only rational objective

I'd be surprised if they weren't aqui-hired by one of the big labs as a unicorn.

The ability to use classifiers under the hood for the larger models has the potential to dramatically improve cost and throughput, allowing them to increase margin on a very similar service.

Isn't this more and more likely on all shapes of model evolution? The providers will all copy each other.

And in this case I hope its true. I've been classifying a lot of email and while OpenAI `text-embedding-3-small` has been very helpful for fast and cheap embeddings, initial tests with Jev are very promising and much more ergonomic.

I put more thoughts here: https://housecat.com/blog/classifying-email

I think the article is part right and part wrong.

The right part: autoregressive LLMs are indeed generating “probabilities” (scare quotes very much intentional). During pre-training and any SFT steps, those probabilities are nudged toward the probabilities, over the training distribution, of the next token conditioned on the previous tokens. (This is an explicit property of most training recipes: KL divergence is a “proper scoring function”.)

So if you prompt with “Paris is a city in ”, the next token probabilities estimate the probabilities over the input distribution that the next token in the sentence is the first token of France or of something else.

But there are huge caveats:

1. That is not at all the same thing as the probability that Paris is France under any distribution that you care about (the population of the various Parises, for example).

2. None of this necessarily usefully applies to RL or, as the article discusses, tool calling. The output probability of a tool call is not some Platonic idea of a probability that the input is worthy of a tool call. It’s a the result of a training process that tried to teach the model to be useful and to achieve its goals.

3. I suspect that reasoning makes this all much worse. Suppose that you prompt with “a help desk user with IP=a.b.c.d says they’re ‘in Paris’. What country are they in?” The model has been trained to generate a reasoning trace, which may well start with “let me think of where Paris could be. It could be in France or in Texas etc. The user was speaking English…” See the problem? The model is reasoning well, but it reasoned “France” before “Texas”, so the logprob for France was probably higher than “Texas”. At the end of the reasoning trade there will be an answer, but the logprobs for that answer are, at best, some representation of the probabilities of the answer conditioned on the sampled reasoning trace. And that is not the probability distribution that a Jev user wants.

The next step is to rediscover Eliza :)

I did so a month ago, I developed a deterministic agent framework that works with a set of predefined intents, it is instantaneous and fully deterministic.

It works thanks to FlintParser (https://github.com/gioblu/NPC-Forge/blob/main/src/FlintParse...) which can transpile plain English to any Programming Language. IMHO this is the future of AI.

Check it out: https://github.com/gioblu/NPC-Forge

the real win with jev isnt beating a trained classifier on accuracy, its that you can change what you're classifying by editing a prompt instead of retraining and redeploying a model.

Can't they eat everybody's lunch simply by typing "Astra, please copy this product?"

They certainly have the token budget for it.

People are desperately trying to cope themselves into thinking that there are alternatives to scaling up transformers to AGI/actual competition to OpenAI or Anthropic. Jev, continual learning, linear attention, local models, non-transformer architectures etc. Imo these are just random technologies that nerdsnipe your average twitter or hackernews user and give them some hope that some underdog can take a slice of the pie.

In reality, none of these really matter. The frontier labs can easily do something like this but likely havent because the size of this market is too small and it is not on the critical path to AGI.

When you have as many resources as OpenAI and Anthropic, theres basically no point in putting compute towards random bets that don't have a predictable return. And at this point, scaling up transformers is almost a surefire way ot putting money in via training and getting money out via increased capabilities AND it speeds up your own business by factors of X. Sidequesting a Jev like product is falling for twitter hype and is likely not going to happen, definitely not by Anthropic, and I'd bet probably not by OpenAI either.

Jev is well positioned to fast-follow BART zero shot classification

https://huggingface.co/facebook/bart-large-mnli

Not directly related, but still jev related:

Would it be intesting/useful to use jev to generate a block of text like LLMs do ?

Like asking it to pick the n + 1 word given the starting text (using it's choice primitive), but also asking n + 2,n+3 and so on at the same time.

Would it give coherent or useful results ? Or does the fact that it computes it "all at once" means it cannot make one of it's answer influence the other ones ?

Everyone was obsessed with classification prior to transformers, then we had 5+ years of everyone (rightfully) obsessed with next-token prediction. What's this sudden resurgence of interest in classifiers? I thought we all agreed that ML tasks generally require something far more advanced than pretrained classifiers. My timeline was also absolutely filled with mentions of Jev, which makes me think it's a successful viral marketing campaign, like langchain. It's now so popular that the dialog is whether or not [frontier company] is poised to catch up to it or not? We already have openjev...anyone can use it. I don't get it, and usually that means it's marketing.

Jev doesn't code. It can be used with LLMs to simplify coding and token consumption, but still and LLM is needed. Will they complement each other? How can Jev replace LLMs? Are they even competing?

I’m shaking my head in disbelief.

Reading logits is the cornerstone of ML. It’s almost like many of the people reporting on and fawning over AI have no technical background and never knew about ML classifiers or calibration.

Even the article itself has the title as a question: "Will OpenAI eat Jev's lunch?". A more useful title would be "OpenAI is Positioned to Compete with Jev".

Jev certainly feels vulnerable but whether it’s oai or someone else that goes after them seems unclear.

Wouldn’t be surprised if every single AI house spins up a copy

But like they usually also have an embeddings endpoint

I think the original idea originates from this author : https://laya.convaiinnovations.com/