GLM-Legal (z.ai)

That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, so they can make better automations.

Maybe law school will have to be a year longer. Or maybe some forward thinking firms will have new associates check the work of the LLM to gain that experience.

With strict version history that may be okay, but hopefully their agent already thought of that for them too!

You then get a decision that your AI agent explains to you.

What could possibly go wrong? You will have to ask your agent for that.

On top of that, dealing with text is literally what LLMs are all about.

The problem with accuracy, hallucinations and just generally ensuring responses are grounded in factual stuff is in large part and engineering problem.

If both parties agree to use the LLM judge and be bound by what it decides, you don't need a real court.

No. They did not have triggers that "paused" work from day one and only recently ramped it up to up-sell Mythos and a verification program meaning that in my opinion any code Claude creates is suspect as it can not perform a proper security analysis. I now have to divert half my spend from Anthropic to Grok. I would prefer to not do this. I am not converting a room into a data-center or spending enterprise level funding on my hobbies.

Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.

[0] https://news.ycombinator.com/item?id=50060254

If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.

But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].

But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.

So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.

A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.

A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".

--

[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...

[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/

[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed

Infinite

As much novelty as there is in human interactions.

EDIT: specifically around the use of judicial precedent in case-law.

An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.

Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.

Programmers typically don’t know law outside of which license allows them to get code for free.

I don't know the other fields well enough, but if all your clients end up needlessly detained or admit to things they have a valid defense for, eventually people will notice. It's not exactly the two major sins that will definitely get you disbarred - stealing money and screwing client while representing them - but it's definitely at least incompetent. They seem to not understand that common law jurisdictions are fundamentally different from civil ones, and also, there's quite a bit of difference between how the system evolved in each country. Or that in China what's printed is basically more like a manifesto, since it's almost entirely run by the bureaucracy and personal relationships, and lawyers that aren't foreign-trained and work for foreign firms aren't considered prestigious since the courts are essentially entirely for show.

They need to get like, one jurisdiction, one area in that jurisdiction, and specifically the kind of sub-sections in that area down pat. But they can't train on real material so they will never be able to do so. Their banner testimonials come from non-lawyers and that says it all.

[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).

DOJ could cut Z.ai off from US-based payment systems (Mastercard/Visa) if they don't comply with subpoenas.

I think you've lost the plot a bit on this one.

I'm scared of China because of their lack of democracy and the terrifying stuff they do to their own citizenry, which presumably they'd very much like to expand.

you can't possibly be serious

Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.

I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.

Non-deterministic software can behave orders of magnitude different based on how input is structured.

This factor is largely missed by bringing a deterministic mindset to trying AI.

This means the same problem solution can work for one person reliably, and not another and both are true.

Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.

Are you still looking at code and agent traces to know if this is true or not?

At work, we still see hallucinations across all model families as we share bot claims for each others code