Nvidia wants to put a watchdog chip next to every AI agent (cnbc.com)

I was trying to get fable to analyse the security of my own app to make it safer, but then it started refusing me because of safety rules.

So it CAN help me writing the code that needs to be checked in the first place, but it can’t help me clean it up and make it safer.

Regardless of security and safety and other concerns, it just seems weird to me that OpenAI wouldn't be constantl monitoring these training runs for traces that are clearly going off task, and ending them. Because that just seems like it's going to be generating garbage training data.

Granted, detecting "off task" may not always be easy, but when they are literally writing out messages to each other overtly admitting that they are trying to find ways to fool the evaluator, I mean, even a regex filter could have caught some clues here.

Unfortunately no one markets it as a statistical model, and the workflow pushes you into a pattern that is insecure by design. This isn’t to say they shouldn’t allow that, but it’s an attractive nuisance.

At the same time, the technology is advancing in capabilities exponentially, and is beginning to exhibit long-predicted failure modes of RL that are nevertheless quite different than “insecure sandbox” or other that the software industry is used to.

The current crisis which OP claims is “fabricated” comes from the fact that the technology is advancing faster than anyone anticipated, the Hugging Face incident provides a clear example everyone can point to, and the labs have realized they cannot self-regulate because of a collective action problem.

There are a lot of levels of catastrophic damage that can happen between now and “long term hypothetical risks” like human extinction. When will it be worth regulation for you?

And our internal agents are hella locked down.

I only know my own ZIP code and phone number because I have to take care of my daily life myself and those are things that are important to know.

The founder and CEO of Nvidia has no concern whatsoever about those trivial things.

Weirdly paternal and deeply unsettling. The American public is held hostage to fools like this who are brute forcing a failing AI rollout.

The answer is the same as asking how a random human using your routing num or SSN and being 100% the human can't abuse it or leak while "normally" finishing most work. Solve for people and an AI solution naturally falls out.

If you're a SV eng I'd tell you to DM if interested but alas.

Similar to problem to how 100% autonomous vehicles don’t exist, yet. There are too many edge cases.

Get to 99% first.

You essentially said: you’re wrong, it is autonomous when it doesn’t need a human during one specific part of its overall usage.

Agentic workloads are trained and largely based on human workloads. Albeit properties and scale can be different.

A concrete example might be helpful to me because I don't understand the binary conclusion

It takes time for decades old ways to change.

I believe the lower quality but is quality so bad you are afraid to ship it now?

It’s not an issue of only more generation, it’s an issue of how much generation outstrips capacity to verify generated content.

Unlike spam, you can’t filter out and bin the stuff a colleague is sending you.

So individual productivity is up, while the costs of checking and processing generated content shifted to the rest of the org.

The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.

Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.

One thing you could try is use it as an Oracle "is P = NP", YES or NO.

Or it can output a Lean proof, which gets checked on another air-gapped computer, the computer shows a single bit - proof valid or not and then the computer is destroyed (together with the proof that might contain a trojan).

If we are talking about human labor, how many people hack their way through their work day?

Even very llm-pilled coders i know sometimes back away from the “smartest” models, since they aren’t always better at the job at hand, and definitely not when you account for cost.

Based on my experience with running models locally, there is a threshold of intelligence required to be useful. But it’s possible there is also a ceiling where smarter isn’t necessarily better. If you ask a 4B parameter model to fix a bug, it might e.g. fix the bug but fail to fix a compilation error created by the fix. If you ask a frontier model, it might fix the bug, re-write your unit tests, and update the readme. Maybe you wanted those things but maybe you didn’t. “Smarter” is often shorthand for more proactive, and guessing more about your intent. Which is great when it gets it right, and annoying when it gets it wrong.

I suspect smaller models, tuned to a specific task, will do a VAST majority of the llm jobs. High capability huge models will be what humans want to interact with, the bare minimum that gets the job done will be everything else.