Agent memory needs its OpenTelemetry moment (lorekit.io)

108 pointsby mthines7 hours ago78 comments

armchairhacker 5 hours ago

Is this a significant problem? It seems like there are 15 solutions to every AI “text format” problem, but all are merely structure over feeding the AI informal English, structure which I believe the “transformer” model is specifically able to strip to a common form (which is how training it on the entire internet worked).

AI does benefit from learning to use skills, and being instructed with good structures (like “save [specific important kind of detail] in memories” and “plan before implementing anything”). But none of these memory solutions seem better than any other, they’re basically just different folders. Your CLAUDE.md can say “see AGENTS.md” and AGENTS.md can say “write memories to .agent/memories/“, and I’d be interested in a measurement, but I assume it performs almost the exact same as native CLAUDE.md and .claude/memories.

One thing I’ve noticed AI does much better than me, is not have OCD over the “perfect” solution.

reactordev 4 hours ago

This is what happens when you treat the LLM as a black box, you end up building the same stuff around it. Like trying to make your dog understand your intentions. You can train it to do tricks and it recognizes specific reward patterns but most dogs have no idea what you’re saying beyond basic triggers.

soraminazuki 3 hours ago

It's sad what software engineering has been reduced to for a lot of people. It's basically pickup artists compiling a list of "body languages" for LLMs, pretending as if there's science behind any of their interpretations.

yladiz 2 hours ago

At least body language has some studies put into understanding it.

soraminazuki an hour ago

That may be so, but most "body language tips" circulated online are cringey and lack the rigor and nuance of proper scientific studies. I think of human reactions as more complex than what those kinds of online materials imply.

It's even worse when applied to LLMs, which are more unpredictable.

speedping 6 hours ago

Entire blog post reeks of LLM. If there was an adequately reliable way of detecting LLM text i'd build a browser extension that would just filter it out altogether.

jaynate 5 hours ago

Thought the same thing and was surprised how high this was ranking. I wonder if we are starting to look past the LLM stank and into the core of the idea (which is solid in this case)? If HN, a highly discerning crowd, is able to look past it then many other audiences are probably okay with it too

grey-area 5 hours ago

Often people don't even read the article, they just see a headline they agree with and upvote it.

I usually read the comments before deciding if the article is worth my time.

Caracas288 5 hours ago

That would be great. Or, for the physical world, combining some kind of smart glasses with the new Pangram image detection which seems to work well on rephotographed data. Like the _They Live_ glasses.

mthines 5 hours ago

Hey, author here. I wrote the blog post and of course got assistance by AI to rephrase sections, gather data and do comparison, and make graphics - also because I'm no native english speaker and wanted to improve the writing.

The sections, points, ideas are by me :)

grey-area 5 hours ago

I imagine you probably wrote an outline with the help of Claude on this topic, then got it to generate the article including illustrations based on that, gave it a look over then pressed publish? Did you write any substantial parts of this yourself?

Kwpolska 4 hours ago

No, you didn't. Claude wrote the entire article and the entire website.

1123581321 3 hours ago

As a native English speaker, please believe me that it doesn’t read well. It reads like Claude or another LLM wrote it. Not only is it Claude’s voice, but the outline of ideas is hard to understand. It’s not a good idea to generate text at a level above your mastery of the language because you won’t know what specific feedback to give. (Similar to the problem with vibe coding and then trying to make it production grade.)

ulrikrasmussen 5 hours ago

Exactly, and LLMs cannot describe architectures, period. On the surface it looks coherent, but everything is vague and lacks rigor. Instead of defining concepts and giving them a name, LLMs keep referring to them using vague terms such as "address and key is where the memory is true". What the fuck does that mean? Give each concept a name and explain them.

This has no value because noone can separate what was designed by a human from design details that the LLM made up on the spot. That's fine for one-off generated stuff like web pages, but as a proposed standard for everyone to build on? No.

Muromec 4 hours ago

It's always doors and seams in my case. Ups, downs, peers, whatever but actual terminology used in the spec. It also has the writing guide with examples of good and bad, but it always needs a second pass to actually follow it.

aacid 5 hours ago

At what point we just start hating LLM because it is LLM?

Honestly I do not care if article was written letter by letter by human or written by agent steered by human.

I care about knowledge and quality, it it is there then it doesn't matter how was it written. I too hate LLM slop but I understand that not everything done by LLM have to be slop.

jaapz 5 hours ago

I think many people are just fed up with the specific voice LLM's tend to write in. Doesn't help many of us now live in a world where most of what we do is read LLM prose in claude code and what not

grey-area 5 hours ago

At the moment it's a good proxy for quality and intention. IF it was mostly generated, it's probably mostly meandering bullshit which doesn't quite make sense.

Yes the style is annoying and bombastic, but the objection goes deeper than that; it's annoying to have someone generate text with an LLM from a small outline or prompt, not even edit it for sense, and dump it on readers.

Muromec 4 hours ago

>At what point we just start hating LLM because it is LLM?

At the point where you read enough LLM written text with zero substance narrating wikipedia article or heard that voice on youtube restating the same thesis over and over for 15 minutes.

The rule of thumb: if I can detect it's an LLM without trying to detects it's an LLM and the feeling doesn't go away, it's a slop.

stein1946 5 hours ago

> If there was an adequately reliable way of detecting LLM text

Yes it is and it's not a technical one: https://digital-strategy.ec.europa.eu/en/policies/eu-icons-l...

hbcdbff 5 hours ago

To point out the completely obvious, the approach described in your link requires compliance on the part of the author.

Given that the author could already have just inserted the text “hey I generated this via AI”, I see absolutely no reason why they’d go to the effort of adding an “AI label”.

nicce 5 hours ago

There is no reliable way. It is like saying that LLMs can only produce output that is tightly constrained and user cannot control it to go outside of these constrains. If that would be true, these companies could not even dream 1 trillion valuations.

Muromec 5 hours ago

If user bothered to control the output enough that it doesn't reek of LLM it's okay to not mark it ass llm vomit.

Squarex 4 hours ago

As an european, this screams classic EU move for me. It adds bureaucracy for the good actors and do nothing to the bad ones.

verdverm 2 hours ago

The question is how do you detect those who do not comply?

SuperV1234 5 hours ago

Hopefully there's also going to be a browser extension to filter out people complaining about LLMs.

zaphirplane 5 hours ago

And filter our LLM slop and sock puppet replies. Looking forward to the clean handful of comments left

holoduke 5 hours ago

Four different ways, five approaches. Three statements and nine topics .... Every ai slop text has this endless useless crap.

acron0 5 hours ago

This is such a thought terminating cliche at this point. Is it possible for you to comment on the content of the article rather than just informing us you didn't like the format? It's not helpful

heikkilevanto 4 hours ago

I would rather have an extension that blocks all the HN comments that complain about a post reeking of LLM. And probably also posts like this that complain about the complaints about the complaints...

brink an hour ago

Hey, you either write blogs using LLMs or you get steamrolled by people that do. /s

Fordec 7 hours ago

On the general premise, I agree. But I don't think that moment is today, not yet anyway. I don't think we understand the right way to approach context and memory yet or how long they should last for, never mind standardizing on an abstraction prematurely that ends up being like SOAP or SGML.

andai 6 hours ago

What's better than SGML? What would XML an HTML look like if they were based on that instead?

moron4hire 3 hours ago

HTML is SGML. You know that while doctype things you're supposed to out at the start of the document? That specifies the document schema of HTML at a specific version for the current SGML doc.

andai an hour ago

Yeah, hence my question. What would HTML and XML look like if they were based on this thing that's apparently superior to SGML instead of being based on SGML?

dizhn 6 hours ago

Fully agree. Any solution people think they have today is working on assumptions that are probably not correct. And things are still moving very fast.

baalimago 5 hours ago

Do we really need this..? What LLMs do well is interpret natural language. So it's a sponge for whatever format is used in a specific project: a predefined format is not really needed.

Besides, do we really want memory of user A's agents to co-exist with memory of user B? Not really. So the argument for collaboration and sharing is quite slim, at least for coding agents. The repository specific context can be encoded into a docs/ or architecture/ directory, just like it's been done pre-agent times as well.

For monitoring deployed agents: maybe. For this I built (yet another) memory management tool slivingdoc.dev, with this it's possible to pull down the durable context of all agents. But here the problem is a distributed system one, not format of the memories.

verdverm 2 hours ago

Structured metadata like this is for the non-LLM tooling layer, primarily your harness, so (eg.) it can interpret multiple formats without needing to burn tokens every time.

imgyuri 7 hours ago

The need for a unified format or spec is greatly reduced in the case of Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats. Even if such instructions did not exist, agents are extremely good at traversing files to get the necessary context it needs.

jstanley 7 hours ago

This is true and it probably is actually fine, but in a way you don't want it to be wasting its reasoning abilities and context window on figuring out which text files to read.

It would be like trying to accomplish something as a human but all your reference material is always in a different format, it would be maddening.

troupo 6 hours ago

> Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats.

Only it's not read by agents, and largely ignored.

From Anthropic's recent self-congratulatory article:

---start quote ---

Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence

...

It loads only the first 200 lines or 25 KB of MEMORY.md

https://www.anthropic.com/research/components-of-agent-memor...

--- end quote ---

mmasu 5 hours ago

I am curious about what people do these days and how do they tackle this problem. At repo level you can alter by with AGENTS.md and perhaps a curated set of additional docs with the project history, but as soon as you get out of it and want to have additional context the solutions available are too many - Hindsight, Graphiti, and many others; or simply roll out your own? I would like something I can spin up with Postgres or with an S3 bucket and that writes me .md files while retaining some sort of relationship among different entities - however it looks like there is no “simple” solution out there yet, with many of the existing implementations being way too elaborate and/or requiring exotic DB solutions. Would be very interesting to hear if someone knows of a bare bones solution I can plug into any agent.

everforward 5 hours ago

The simplest I’ve found is to add instructions to the AGENTS.md telling it to keep a knowledge base in a directory and refer to it. You could add a hook or something to use the AWS CLI to sync the dir to S3.

The agent actually did pretty well managing and searching it with grep.

A lot of these memory systems are reaching for the stars and I sort of suspect we’re way past the Pareto point for coding there.

skeptic_ai 5 hours ago

Why not a simple git you can push to main directly?

everforward 3 hours ago

The notes and code are usually tied together, and putting them in separate repos just forces you to sync them.

The notes for v1 probably don’t apply to v2.

Maybe different if you’re doing a global memory instead of a per-project memory.

Muromec 4 hours ago

Right, I have a simple key-value thingy backed by the filesystem. Not even gzipped. Maybe I will get something elastic shaped or what not at some point, but likely not.

Like what are you store there that text files aren't enough.

totetsu 6 hours ago

What about .. reference identity.. currently memories and recall are referencing local things, that are specific to the environment they were made. and in ways that are as context eficient as possible like @message100Line10-50 or sessionsabc@100-300 is ./yourfiles/thisfile.mb ..

but is were going to make the memorys portable.. we need to make the things they are pointing to portable also..

locallylinear 4 hours ago

This is interesting but also think about how we as humans memorize things in code

I don’t think “oh it’s lines 10-50 in file A that has this code I’m looking for”. It’s much looser, something like:

- maybe I can’t fully remember file A’s name but I can fuzzy search for it based on what I do remember

- I search for a function or variable name that I remember, again kind of fuzzy searching. Or if I can’t find it,

- I start quickly scanning the whole file for relevant variable/function names

Obviously storing exact references will be more efficient at lookup time but to your great point, the environment is changing too much, especially now. Having just a) a starting point on where and what to search and b) an understanding the code base will and should change , feel like some of the principles a memory system should assume. Not that it’s mutually exclusive with what you said - just not sure the juice is worth the squeeze to make the thing memory is pointing to portable.

totetsu 3 hours ago

I guess it depends of the use case. For episodic memory that need hard evidence, like .. what did we decide last week.. it would be good to have the link to the original document to look, rather than just an extracted text.. Especially if those memories are going to be subject to reranking or "dreaming" or some non-deterministic clean up .

htsh 5 hours ago

I have been running openviking self-hosted for 6+ months now and it has worked well for me. It is open source, self-hostable, and you can bring your own LLM and embedding model.

The bytedance team makes it. I often wonder if it does not get attention here in America b/c it is Chinese?

grey-area 5 hours ago

Should we even be storing 'memories' in text files at all?

Imagine how humans would function if we carried our memories around in overflowing notepads and had to consult them every time we wanted to know a fact or remember a person we had met, it would be exhausting and inefficient.

Muromec 4 hours ago

It would be nice if local models will unlock the way to put memories back into the model itself somehow, but it's not big labs who would bake something useful here I'm afraid.

hagen8 7 hours ago

Well, coding agents are smart, they can use all kinds of memory. A simple skill on how and when to access it is enough...

eladlev 5 hours ago

How does this handle parallel sessions writing the same memory? From a quick look at the repo, it seems that one session can silently overwrite another. Anthropic's recent memory post shows how severe this can be (https://www.anthropic.com/research/components-of-agent-memor...). Should the spec cover this?

Muromec 4 hours ago

Maybe don't give all your agents the same memory. Shared stuff can be in the repo they work on as files.

hemkeshr 6 hours ago

I agree this is very much needed but I doubt the other ai companies would a-comply with this as they need users and mainly paid users otherwise they'll lose themselves. What I think is that with 20$ subs of 2-3 ai can give you more usage compared to their 100$ subs if you do it that way. I don't know will these companies work together to form a standardized memory system. I highly doubt that.

mohsen1 3 hours ago

Working with LLM agents comes with a hard pill to swallow: they are already remarkably good at figuring out what to do on their own. In fact, the more rigid structures we impose on a system, the less efficient these agents become.

While there have been many attempts to structure how agents operate, a recent Anthropic article showed that when they didn't tell Claude how to use memory, it actually used its memory better. Ultimately, we have to come to terms with how little is required from us when LLMs are this intelligent. often doing things better in ways that we sometimes cannot comprehend.

mceachen an hour ago

I don't buy it. You can steer them gently to at least write human-accessible project plans—and then session compaction, agent switching, and review is much more doable. I've been iterating on this for over a year: https://photostructure.com/coding/claude-code-tpp/

rickdg 5 hours ago

I'm ready for the conversation to just assume a linux file system with mostly markdown. And you schedule some time every day for an agent to make it not suck as much as a living archive.

What are some real issues now? Auth? You want to retroactively implement boundaries? Is version control the problem?

totetsu 6 hours ago

I think It needs to be as diverse as people .. If agents are to become our constant "muse" like companions, I want the wiring of the brain of mine to be as wired as possible.. Why not juggling chain saws in orbit.. https://www.youtube.com/watch?v=JcJSW7Rprio&t=17s

jaynate 5 hours ago

This seems like local memory for one user. The need for corporate agent memory that works across users and company processes is acute.

kohlerm 5 hours ago

Yes, also I doubt that we already know how to best implement memory for plans. Is it really only files? Maybe some db approach with queries is needed?

slopinthebag 7 hours ago

at this point i'm pretty convinced agent memory is strictly harmful. it remembers the wrong things, and then turns them into an inane focus for each task. i disable it universally on every harness. if i want memory, i specifically instruct the agent to record findings into a file.

SwellJoe 7 hours ago

Same. Everybody builds one, and then eventually realizes it makes the model stupid and confused. The right solution is pretty much always developer documentation in the repo, in roughly the same shape you'd make developer docs for humans that you want to be able to get up to speed about your codebase really quickly.

This has the benefit of being something you can easily read over every now and then to make sure the model hasn't baked in some random comment you made about a specific circumstance a month ago and made a weird religion out of it.

nonethewiser 6 hours ago

I strongly agree. This is becoming a general principle along the lines of “code comments that describe what the code does are bad.”

Memory just ensures there is a fairly high level of drift and inaccuracy in your context. Unless YOU are manually curating it.

Its similar to wiki systems where your agent produces tons of documentation and decision records, etc. Its a bunch of assertions about your codebase that you aren’t vetting and which is at most 90% correct. Thats good enough for a feature or bug fix that you can validate with tests but for the love of god throw the details away. You don’t want to codify that. It’s polluting the context and gimping the model.

nonethewiser 5 hours ago

Also introduces massive confirmation bias. Majorly gimps the model. I remember I was exploring hexagonal architecture on chatgpt and then future conversations with open ended architecture questions just gave me those same patterns.

“Oh damn look its recommending something that looks like hexagonal architecture. I guess hexagonal is a really good - hey wait a second …”

Kwpolska 4 hours ago

Agreed. Claude Code often tries to write one-off decisions and transient problems into memory. And when such things appear in the context, Claude will start doing questionable actions, because it saw something unrelated and wrong somewhere in its context

cryptoboy2283 5 hours ago

Do you mean another overcomplicated enteprise standard teeming with constant CVEs to increase surveillance and control?

te_chris 7 hours ago

So we need another complicated format to store information so that the machines which read unstructured text can work with it?

jarym 3 hours ago

Maybe I’m going to get downvoted for this but premature-standardisation is a problem.

We only have OpenTelemetry because of years of people working out what metrics and logging look like.

It’s too early in my opinion to standardise agent memory.

jonstewart 24 minutes ago

Hacker News needs its slop-rejection moment.

j45 2 hours ago

Some times when I see ai generated text like this, it feels possible to see how sophisticated their AI skills are if they couldn’t even get the text to render better, and how that might transfer to the code itself.

This looked interesting, unfortunately it feels far too long for humans to spend time reading.

People might really be starting to write like LLMs, it’s why reading books is the single biggest anti-slopping protection for the human brain.

The “shape” of the article was a word generator performatively generating larger words where simpler ones would do. “That holds”

mfkhalil 6 hours ago

Not trying to be disingenuous but what's wrong with just a normal filesystem?

The best version of memory I've experienced is when I give my agent a memory directory and just let it do what it wants in there, with minimal rules on my end.

alfiedotwtf 6 hours ago

Files aren’t indexed and vectorised by default

dizhn 5 hours ago

This is true but is it proven with reasonable confidence that indexing and vectorizing the code provides better results. I would assume it would if another agent is responsible for it and your coding agent only talks to that agent. Even there I am not 100% sure. Probably not worth the effort though.

monster_truck 7 hours ago

it's called a markdown file

hbcdbff 5 hours ago

This is slop. If you can’t be bothered to even edit your AI slop, I won’t be bothered to read it.