Introducing System One Models and Jev (typesafe.ai)

My first thought was that it would be ideal for robotics? As in control of limbs, general planning, route finding, etc.

> Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.

that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do

And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.

I foresee this to be the path moving forward - giving AI models understanding of the computation directly(as well as compositional rules) This feels like a short path towards total software in many areas.

The comparison between LLM speed and Jev speed is misleading, because they're using autoregression to generate all of the type names, all of the schema, etc. A closer comparison would be if the LLM was purely outputting the raw numbers. Even then, comparisons to LLMs are pointless because you could train a transformer on the same sort of task that Jev is doing and get even better performance yet again, and a smaller model. I suspect this is some form of stripped down diffusion language model.

You really have to do a lot of hand holding here, and map out your problem space manually, and very carefully, to get any sort of accuracy. For example:

> Keep each Score to one dimension. If a description says “punctual and smart and experienced”, the question is measuring three things, and an input that is high on one and low on another can’t be placed. Confidence drops and the score means less. Split it into one Score per thing and combine them in code

If you don't perfectly represent the distributions of possible answers then you'll likely get garbage results. As far as probabilistic state machines are concerned, I'd say creating the distributions of possible answers, and their hierarchy, is the actual hard part.

One of their examples is:

- "state": "I have asked three times now. Can I please just talk to a real person?"

- "Is the customer asking for a human agent?"

Imagine the users request is: "I want your human agent to call me tomorrow at 5pm."

Human conversation is fuzzy, getting useful reliable results out of this is going to be a challenge. Of course, you could add follow up checks like: "Do they want that now, or later?" -> if later -> "Do they want that tomorrow, or the day after?" and so on... But now you're building an LLM out of if statements. I am skeptical of whether this model has much utility for fluid language interpretation - I suspect it'll only be useful for scenarios where you've tightly constrained the answer space but want to use fuzzy language to describe it. Like:

- Question to human: "Would you like a support agent RIGHT NOW?"

- Their response: Yes | Yeah | Mhmm | ye sure (any possible yes signal)

Model input: "Did they ask for a support agent?"

Still... a tiny LLM could accomplish this sort of thing without problem. And that doesn't stop someone from saying: "No, not right now. But tomorrow." - and the tomorrow would get missed. I think this is why people haven't really tried this approach much already.

Also their Doom demo is on structured state, not on images. Meaning, the enemies must be being served to the model as coordinates (or the exact angle of projectiles that hit the player), otherwise it'd have to scan every pixel of the 360 degrees to know whether an enemy is in front of the crosshair or not. You can see from the map below that it's also choosing travel checkpoints/destinations through walls. So they've severely cooked this to make it look far more capable than it is in practice, and any speed advantage that is offered here is not factoring in the shortcuts it is taking, the training on the map, and the fact that it can cheat because the structured state it is using is not bound by obstructions.

Here is their docs by the way: https://docs.typesafe.ai/ - so you can understand how it works.

For all we know this might be a non-language-generative transformer e.g. a transformer where the decoder produces confidence scores rather than language. Please provide more likely architectures if you know them, I'm genuinely curious.

So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).

To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.

(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)

The Doom demo seems very funny business. They're not feeding it video, they're feeding it a text description of what's going on in the game. It's not reading pixel data.

I think LLMs would play a lot better with that input too but Jev does seem to have a huge speed advantage; I don't know if the other models could do that in real-time.

Side note - just like most people don't need an intelligent personal assistant to manage and respond their emails and book their flights, most people also don't need smart homes. Century old toggle switches are more than enough in a 3 room apartment or 5 room house unless you have a mention.

  from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
  
  with TypeSafeClient() as client:
      response = client.system_one(
          state={"document": "I was charged twice. Please fix this ASAP."},
          questions={
              "billing": Noul(instructions="Is this ticket about billing?"),
              "tone": Choice(
                  instructions="What is the customer's tone?",
                  criteria={"calm": None, "frustrated": None, "angry": None},
              ),
              "urgency": Score(
                  instructions="How urgent is this ticket?",
                  criteria=["can wait", "this week", "today"],
              ),
          },
      )
  
  print(response.nouls["billing"].noul)
  print(response.choices["tone"].choice)
  print(response.scores["urgency"].score)

I need to ...? -> Open-weight model.

I'm sure someones working on this as we speak using an open-weight LLM base (Qwen or something would be a perfect fit).

This sort of task is a perfect fit for a very small model capable of semantic parsing. You can get away with a LOT less parameters without all the autoregressive generation and long-context reasoning.

I do agree that the comparison to LLM tokens is hard to understand (also because output tokens are not comparable).

But yes, text or structured state (like a JSON with multiple pieces of text in) -> decisions out (e.g. choice maps to "match" statement, "score" maps to sorting, "noul" short for bernoulli maps to if-statements)

Input, and criteria/instructions can both be defined as structured input (JSON). This ends up being pretty powerful because the model is trained to understand structure.

e.g.: https://docs.typesafe.ai/primitives/advanced#structured-inst...

> not sure if multi-modal

just JSON... for now :)

> outputs the question's answers as appropriate

correct!

this is complex, but generating text is highly complicated and requires mode dropping to make long cohesive text

So, assuming this is not vaporware, this would raise the tide for everyone because it shows what's possible.

I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?

But agreed it'd be very useful to see it deployed on other hubs, and it seems worth it to provide the bespoke API format. Perhaps Typesafe's API will end up becoming the standard for a new type of structured model, the way OpenAI's API did.

It's totally reasonable to compare against LLMs doing chain of thought if it gets comparable performance.

I think this is reasonable if people are actually using LLMs to solve this type of narrow structured task, which they are. The evidence is that every LLM provider has some method of forcing the output to conform to a json schema in their documentation.

Their manifesto: "you only build on top of it if it's trustworthy." - the irony of this while putting out the most misleading, dishonest marketing campaign I've seen in months for their first public appearance doesn't exactly scream "trustworthy" to me.

constrained decoding (OpenAI-style structured outputs) make models dumber unfortunately - the short+dense version is that simply masking logits is insufficient because if ever a model was assigning probability to an invalid token, the model is by definition confused. you'd be better off erroring IMO

While this model may share much with GPT-style models on the encoder side, it clearly has a different decoder architecture. So is a high-parameter count language model an LLM even when it doesn't have a GPT-style decoder? The definitions are in flux.

you pass in your "prompt" and options (described in natural language) that it can respond with, in addition to your input. it gives back that option set with a probability assigned to each one