Postgres 19 is late because so many more bugs are being found now (rhaas.blogspot.com)

Because they are large codebases with their own technical debt. You're not doing a full rewrite, but just make what you need for you own codebase.

Actually, I have started with it already but is paused, but the approach is to rebuild it from scratch in parallel to our current legacy system and move our staff over to the new completed screens in batches (we have close to 1000 views in our system, 150 background jobs, with about 15 main categories (think organisational sub-divisions)). So as long as we go slow and steady (even with AI) it will work out fine, I expect it will be done in less than 6 months. One month if we rush but we don't want to rush it, as we are moving some things over from MSSQL to Postgres, so slow and steady is the way.

We are multiple developers, each having 10+ main claude agents running different sessions/terminal tabs. I have 10 copies of our codebase on my laptop, each with their own purpose and claude session and history. The other devs have different approaches but we all get similar output. And then we also built an agent framework that runs within our system that has access to our code (at most 60min behind master branch), our infra and our databases, so it can diagnose issues with excellent precision and prove it to us where to find code/data issues.

Myself specifically, I have a /code folder without about 10 copies of our codebase, each running their own claude sessions & git branch. Then, I have 10 terminal branches. I spend a ton of time planning out & specing the problem with claude, to be as specific as possible and be as constrained as possible so it cannot go on side-quests. I use Opus 5.5 on High effort. That's just on my laptop with what I can mentally cope with day to day. Then we have some other automatic ai goodies running within our custom agent framework that runs within our platform that also helps. Then for apps, I basically plug in multiple phones into my laptop with usb (and emulators if I really need to), and claude can fully drive the ui and testing on the phones via adb as an example. Even on linux/fedora/kde it can drive UI apps a and take screenshots and fix things properly etc. It can ingest all logs & debug chains, so it absolutely flies through its task.

Before we started this journey (about a year ago), we spent a ton of time cleaning up our codebase, splitting large files (10K+ lines) into files that are no more than 2K lines, things like that. Then we let claude scan our important files and summarize them, add comments for future claude sessions. So we've let it built up tribal knowledge per file and the reasons for things to exist. We tell it all about our operating environment, the country and legal framework etc, so every little piece helps it stay honest and on track.

So to start, do something simple like: Scan the files in this folder for database code and check for queries that loads the entire result set into memory and/or queries that loads all columns. Suggest which queries we can filter better or reduce columns being transferred. Once done, also check real db row counts, compare them with the queries in these files and check if they might benefit from an index or partitioning/sharding. Also check for queries that loads data but their results are not used. Don't make changes yet, just do a deep investigation and report back. Feel free to spin up muliple agents to batch the scans.

edit: it goes without saying you need a language with strict typing and a good compiler (we use dotnet). Doing this on python/javascript would be a nightmare.

I personally started using AI with a very rudimentary "memory" system, and for first weeks corrected it ad nauseam, until, in the small domain I was responsible for, it started doing work end to end fully autonomously. Takes a lot of time and effort to bootstrap, but once steered well enough, it just clicks and less and less is needed to check manually

Our legacy system runs mvc view with bootstrap 3 and some jquery. It's not a SPA, so no framework to replace. But we still have to mess around with npm sometimes.

We haven't fired a single person yet due to AI. We just save everyone time, make input more accurate, make our accountants better, manage our technicians and jobcards better and so on. We also made our system bullet proof to the point that we only had one weekend failure in the last 24 months, and it wasn't our fault (a third party dependency did a large deployment on a friday night). Our system is now deeply "boring" and its awesome.

Perhaps the ultimate use case would be to use AI to scan, debug and find issues etc, but never to generate new code. Still have humans writing and understanding everything, keeping things as clean and concise as possible.

The only companies who will really need to fire are the big tech companies who hired just because, for empire building, or to keep talent out of competitor's hands. Nearly everyone else is actually positioned to leverage AI to now compete against the bigger players. I'm already seeing small companies have a big surge.

This is how I use it. I don't run any companies (nor am I a part of one right now) but were I to run one I would employ AI as the ultimate bug/vuln hunter. Just spin up maybe 20 agents and say "fuzz this thing to death".

> Now this is how companies should be using AI. Not as an excuse to fire half the devs.

AI is getting so good, it will probably be both.

When you change a single setting in the IDE to make something more strict and you have 1M+ lines of code, you will get 30K items to fix, via red/yellow/blue squiggles in the IDE. Most of those can be simple refactorings, but the performance & UI of the IDE makes it incredibly unwieldy and it will sometimes fight you back, sometime crash internal processes which leave refactorings half baked and so. Now imagine you change 50 such settings, to make the compiler force you to write better code. Changing 200K+ lines of code yourself is a nightmare (and to prove it still does the same thing afterwards). Instead, claude systematically approaches whole classes of errors and convert them to cleaner variants (either using our custom skills or it writing new scripts to do it). I consider myself an advanced user of IDE's, know how to do bulk refactoring and renames and so on (esp with JetBrains tools), and I would say it would be a near impossible task to do as a human. Then there are other developers - so you flip a setting and now 5 other people's IDE's are forcing them to work a specific way which will make you unpopular very quickly and revert the changes. So, since most of us no longer use IDE's as our main tool to build/diagnose, we can more easily flip compiler settings without destroying someone else's week. Hope that makes sense.

Oh it also helps to split refactoring work into three buckets: business logic changes, performance changes (same output as before but faster) and cosmetic changes (rename files/folders/classes, breaking large files up, add logging/comments, domain layering etc). We tend to focus on performance and cosmetic changes for bulk refactors and leave logic changes to more dedicated sessions as to avoid changing what the system do (our domain is fairly complex and we cannot afford breakages). We also managed to port ALL of our code to dotnet 10 lts and have clean boundaries between projects (we have a mono-repo with about 50 projects in it), so we can wholesale upgrade dependencies without a sweat.

I don't think it'll be that bad:

1. The cost of running AI is dropping down to "free" - there's open weight models, open source tools, and guides on running your own if the long list of suppliers isn't good enough for your use cases.

2. Vulnerabilities have always been a risk; AI is not a panacea. The pre-AI best practices still apply. Furthermore, a huge percentage of vulnerabilities are in 3rd party tools and systems, which are systematically being scanned and fixed and/or exploited, then fixed.

You mention "the free software community" in your last sentence though - this implies the code in question is open source, so it's able to receive plenty of improvements from anyone with access to these tools.

Short term there will be pain (hacks/malware) but at the same time, it will force the survivors to build bullet proof systems.

Worse case, we enter the Dark Forest era for technology. Read up on this concept and apply it to the tech industry. I suspect open-source might die and everything becomes private, no more public networks, no more open access, as to assure survival. In a sense we are halfway there already as it seems the internet is starting to close down, people not posting their art and music as much anymore, not posting their knowledge for free anymore, as it might be used against them by AI (the other party in the dark forest..). Thus private authentic experiences will become more valuable.

One problem I’ve seen with third-party libraries, especially in the JS ecosystem, is that to fix a CVE, you often need to jump a major version or three, and that entails quite a bit of work to adapt to breaking changes. Porting a fix back on your own and keeping your own fork is the same as supporting a first-party library, except it’s not really first party, and you’re not familiar with its innards.

on the other hand you are not exposed to supply chain attacks

>Avoiding 3rd party libraries seems like a false sense of security.

What a bullshit.

It's different scenario, different trade offs.

You mitigate risk of library being sold to malicious 3rd party, you avoid risk of your supply chain being hacked, you avoid API/behaviour breaks, etc.

The trade off is that you have to maintain it.

That's why dotnet's ecosystem is so loved due to MS providing 90% of needed libraries, so you avoid some 3rd parties maintained by single person.

I understand, but I would argue its better to have an in-house 1K lines of code that only does what we need and nothing else, it well tested and under our control, than to depend on some 50K lines of code (of which we might only use 10% of functionality and the others just getting in the way) that random's on the internet can modify, sometimes pulling in transient dependencies that we would really not have in a server environment, telemetry and other gremlins. It goes without saying that opensource packages will become even more juicy targets for malware/hackers to get into your systems like we've seen in the past month (and it will only become worse - you will be forced to pay technical debt or die).

The only "third-party" packages that we want to ideally allow/support would be provided from LTS versions of our runtime (in this case, packages provided by the dotnet team and audited by microsoft). So we will always try to build around what the framework provides. Second choice is our own code, third choice is open-source packages, fouth choice is a separate application/appliance (lets say ZenDesk for ticketing). Over time, we will reduce this footprint to only have dotnet team code and our own.

Remember most CVE's can be hidden on servers that are not publicly routable and don't have a real exploitation paths. The bits that are public (lets say a linux edge server with a web server on it, some firewalls, a couple of open ports) - these should be hardened to the max and not allow full access to all the services running behind it. We don't even give our internal services public IP addresses to make near impossible to route traffic to them.

Our git repo's are also not on public infrastructure (serious, think carefully about hosting on public services and if you really need it), never mind making the repo public.

We also have a dedicated secrets/certs manager, so no credentials are in codebases or pipelines. Very strict in this regard.

   NAT (particularly NAPT) actually has the potential to lower overall
   security because it creates the illusion of a security barrier, but
   does so without the managed intent of a firewall.  Appropriate
   security mechanisms are implemented in the end host, without reliance
   on assumptions about routing hacks, firewall filters, or missing NAT
   translations, which may change over time to enable a service to a
   neighboring host.  In general, defined security barriers assume that
   any threats are external, leading to practices that make internal
   breaches much easier.

It has been proofed again and again that you cannot trust npm/cargo not even AUR, implement you own minimalist functionality without dependencies and just for you own needs seems to be a much better solution.

I feel like security by obfuscation might actually be valuable in the age of llm attacks

Similar to distroless containers & statically linking all dependencies

I think avoiding all 3rd party libraries is excessive, but engineering is about tradeoffs. In the extreme case, do you pull in a LeftPad dependency or write it yourself?

> Avoiding 3rd party libraries seems like a false sense of security

To be fair, there is more nuance to your statement.

A lot of it depends on your programming language.

For example with Go, with its extensive and excellent stdlib, you would be foolish to rely on large swathes of 3rd party libraries, because so much can and should be done using the Go stdlib.

On the opposite end of the spectrum, you have Rust with an effectively non-existent stdlib.

And the problem you see frequently with Rust is projects importing half the internet, with a Cargo.toml the length of your arm with hundreds of dependencies. Of course the project has never reviewed its dependencies source code, and the project probably barely keeps its dependencies up to date. Its a supply-chain attack waiting to happen, not if but when.

So with languages like Rust, you have to take an approach of "minimal reasonable dependencies", make an active effort to keep your dependency list short and in-control.[1][2]

> less well audited than public libraries

I also struggle with this particular argument because the reality in open source is "well, its open-source, it has many eyes" is in fact mostly a dream. Since we all know 99% of the time its a small group of maintainers (assuming there is are even multiple maintainers[3]) and not many (if any) external reviewers.

[1] https://tweedegolf.nl/en/blog/119/sudo-rs-depencencies-when-... [2] https://tweedegolf.nl/en/blog/104/dealing-with-dependencies-... [3] https://xkcd.com/2347/

Agreed. It’s the modern security through obscurity.

Security through obscurity protects you from low effort vulnerability scanners.

It makes you more vulnerable to a sophisticated targeted attack. A competent hacker or AI will quickly find vulnerabilities in your custom libraries.

I’d be happy to bet with you if I rolled my own log4j I would have never implemented jndi as a formatting argument.

Sure I shouldn’t roll my own crypto but I can write a logger…

Using 3rd party libraries is a false sense of security

> just in your own code that is less well audited than public libraries

why do you think public libraries, which change all the time, get deprecated, handed off and supply-chain attacked are well-audited in any way?

It really depends on what the product is, what the 3rd party library is used for, and what the alternative is. Unfortunately, some people are too biased towards using 3rd party libraries and don't understand the tradeoffs.

For example, a few years ago I removed a rather large 3rd party mathematics library because we were only using it for a single mathematical function that was very easy to replicate.

Remember, 3rd party libraries need to:

1: Be periodically audited to make sure you are in compliance with the license terms, and preferences of your company.

(1.5: This becomes important during an acquisition as someone will always come around doing due diligence.)

2: Be periodically updated (in case of security bugs)

2.5: Be updated on short notice when a security issue is revealed.

In the case of the mathematics library, replacing it with roughly 20 lines of code and 100 lines of tests is a significantly better tradeoff than constantly reviewing it every time one of the above things happens.

Now, at what point is it "worth it" to use the 3rd party library? In our case we'd need to make much heavier use of the library, probably at least 20 different mathematical functions from it.

3rd party libraries are attack vectors and if like many many people you’re only using one or two functions then it’s much better to get rid of them.

If you reduce the amount of code/code paths you reduce the surface area that vulnerabilities can appear on. A code base with 10 million lines can have a lot more bugs than one with 100k.

Two edged sword. That means the bots also have a library of exploits to try for the components used in your project.

You’re probably right about some third party libraries, but not about all or even most of them. Package repositories are full of half baked packages that are the result of a single person scratching their own itch once upon a time and thereafter forgetting about it. The more harmful part happens when those get imported into other third party libraries that give the image of being better supported. Supply chain attacks are the new null pointer.

I personally only rely on 3rd party libraries that are complete. I will never use a library that gets consistant feature updates. I do make an exception for openssl rusttls and whatnot since they have to update by design.

but public libraries can also be audited because they are public

For database programs, I would disagree, this should never become something like the nodejs ecosystem.

The one thing that cuts the other way is code churn. Third party libraries may keep adding features you don't use, and those may expose new bugs. You can pin an older version, but unless the library back ports security fixes, you are not going to benefit from the shared development wrt to security.

If you spend a ton of time hardening your own library code and don't have much feature sure, then that can be a win.

I’m not going to eliminate every single third party dependency in my system, but I’m on my way to eliminating 90%, maybe more.

For me it’s less about bugs and more about supply chain security risk. And with AI and only building what I need from a dep, exactly how I need it, the cost of doing this is very low.

False dichotomy. A library that is not a 100% perfect match for a problem you have to solve, will require glue code. Any bug in the glue code is in your own code but only exists because of the library.

While that is true, there's also the other side of the coin: most bugs are hit, but never reported. Users are pretty unlikely to report bugs and when they do they are very vague about it. That's why client observability is such a boon for teams because they find what bugs users are actually hitting. If the agents are directed using that, then perfect

Based on what?

If it enables faster feature releases and better performance, probably?

AI can help companies provide more features or cut costs.

Good thing is now human does not have to justify anymore. If AI says bugs management seems far more amenable to support fixing. At least in my experience managers take AI reports or vendor tool generated reports more seriously then any feedback from engineer.

No. Technical debt is just an imperfect metaphor.

You never wanna intentionally repay tech debt as slowly as possible. You might be able to tolerate/ignore it for longer depending on the circumstance, but you always wanna get rid of it sooner rather than later.

All else being equal, you're always better off without debt. AI can make repaying the debt very cheap (compared to doing it manually).

> But I’m still not convinced AI assisted development is at that point yet;

But the gp said it is, and gave examples.

It’s pretty clear there are a lot of valid reports based on the large increase in backport commits cited.

likely both

I'd imagine just as every other project, both.

They found a more efficient way to play Whac-A-Mole with bugs, they didn't find a way to reduce bugs.

We could really use a lot more luddites tbh. Doesn't seem very applicable here tho.

Did you honestly need a /sarcasm tag in the article?

The article also concludes that overall, yes, finding more bugs is a good thing. Basically saying it’s the LLMs “fault” that the quality is improving, alongside a slight delay to the release.

Luddites were actually pretty reasonable when you dig into it. Maybe shouldn't be using it as an insult?

Luddism, did not expect to ever hear that word with Postgres development :>

RTFA

> In the end, I don’t think that the delay in shipping PostgreSQL 19 is particularly concerning. It is not a long delay in the grand scheme of things, and it emerges directly from the project’s concern for quality.

Interesting that this could result in net negative software productivity world wide in terms of features right now.

LLMs finding unpatched vulnerabilities faster than they can be reasonably patched, bringing all other software development to a halt.

  If that same city was subjected to weekly riots

  That's productive.

We’ve also been blowing a lot of retainer hours just patching sites instead of building new features. It sucks. Ultimately going to cost clients more.

How did you get product management to agree to that? Ours said "I could fix all of that in 2 days with Claude." They are still not fixed. Granted, our PMs suck.

It’s amazing how much slop humans have produced that now needs to be fixed by AI…

Same here. I got access to Mythos two months ago, it confronted me with decades of nasty bugs I wrote. This has been stressing me out a lot, caused sleep deprivation (the bugs follow me in my sleep, yes, really). All other progress is halted, I've been squashing bugs day and night. I see a light at the end of the tunnel.

Bottom line: it's a good thing. My software will be better after that. Even if it's currently not good for my ego (and my health). But I'm glad for all the tools that help me improve quality.

And sure, I can just ask LLMs to fix the bugs, which works well half of the time, and produces bad slop in the other half. This is where human expertise and judgement is still necessary to avoid letting our code bases drift into LLM spaghetti hell.

I don't see how that's a relevant comparison. These are software bugs. Once you find them you fix them. You can't predict the harm from a small bug when AI can chain complex exploits and when your software is used in mission critical applications as a data store. There's no paradox here, bugs are bad and you want to fix them all.

Doing surgery on a code base is dramatically better than surgery on a person, the analogy breaks down pretty quickly I’d say…

Medicine is fuzzy, software is mostly maths. You can prove your software has no bugs and you increasingly should be proving that. The hard part is knowing if what you prove is what the software is doing.

> some bugs you don't want to find

The key thing you forgot, which makes this approach non-sensible, is the omniscience, without which you have no clue whether your posture towards a new bug-finding tool is good or not, since you don't know whether it will mostly find those bugs that are not worth fixing and the time wasted on fixing them is detrimental.

Similarly, how is ignorance better than informed decision when ignorance also includes missing critical security bugs?

That's a flawed analogy though.

Firstly, the cost of fixing a bug once you know it exists is usually minimal compared to the cost of discovering that bug. This is the inverse of medicine, where the cost of testing (at least in terms of health outcomes for the patient) is typically minimal compared to the cost of treatment.

Secondly, if you have an asymptomatic illness, this does not typically make you more vulnerable (hence why it's asymptomatic). Like you say, you typically only die of one thing, so if you know the patient won't die of a particular diagnosis because something else is more likely to kill them, then it might not be worth making that diagnosis. But for software, you can be simultaneously vulnerable to lots of things at once, and any of these vulnerabilities can be the one to "kill" you. Some vulnerabilities may be easier to find and exploit (or do more damage) but in principle any active vulnerability can be exploited.

Thirdly, having multiple vulnerabilities makes software more vulnerable overall, because those vulnerabilities can potentially be chained together to create more serious exploits than each vulnerability would have represented on its own. To a certain extent this can be true in medicine as well, but generally it isn't.

Fourthly, we can't cure death (yet?) but we can cure all the vulnerabilities that we've found so far. There is no known upper bound on the lifespan of a piece of software, and we know that older code tends to have fewer bugs in that newer code, presumably because those bugs have been fixed over time. This is different from health issues, where a given condition may be chronic or terminal and therefore at a certain point there's nothing you can do to fix it.

Finally, ultimately in healthcare it's your body, and your decision was to what you do with it (assuming you can find a doctor willing to do what you want). If you have a condition and you want it cured, even if it is completely benign and will have no impact on your life whatsoever, then you can do that. (Again, assuming you have the resources and the condition is curable). But a software vulnerability does not affect you as the developer of the software, it instead affects your users/clients/customers. Therefore there are different dynamics in play. If I sell you a tool and I know it could be vulnerable, but I don't try and find out what those vulnerabilities are, then you're ultimately the one on the hook when all your data gets deleted or leaked.

> The key distinction is that people only ever die of one thing and having a tumor that doesn't kill them is often better than risking the surgery to remove the tumor.

But what applies to humans doesn't apply to software. First, to pursue the metaphor, codebase can definitely die/be compromised/crash because of multiple reasons at the same time; second, there is virtually no risk, out of man-hours, in fixing a bug.

> Ignorance of the bug is better than knowing about it and having to make an affirmative decision not to fix it.

It seems like this option is being taken away by LLMs for all major and widely used open-source projects.

For better and worse.

I will add more example links in the top-level comment, but here is the thread for that bug. [0]. tl;dr timestamp vs timestamptz comparisons aren't consistently ordered in a pathological edge case, which means that index scans can return the wrong answer. There's no way to fix it without either redefining the operators in a surprising way or disabling index scans in this case.

[0] https://www.postgresql.org/message-id/flat/c22286ad6268347c6...

I am pretty sure we have always been able to add holes faster than we can find them and fill them, and that this has not changed with the advent of faster hole finding tools that are also faster hole producing tools.

I've been engaged with Postgres community for over 20 years now, and this has been my experience, too, throughout that time. Off and on I have interacted with a few other large projects, and nowhere (except at CodeBlocks) have I found such a welcoming and responsive community. The CodeBlocks IDE project has been one exception where I felt welcomed; but my interaction there was quite limited, and spanned over just a few days, so I can't extrapolate too much from that.

BTW, the Postgres community is also quite protective of their culture, so I expect this welcoming nature of the community to persist for a while.

> The most common error for a search task is omission, and that's really hard to spot

NP-Hard, even[1]!

[1] https://www.i-programmer.info/news/112-theory/3830-physics-i...

The planning, which can also be done in concert with AI, before getting the AI to write the code helps with not losing track of what you're building. In my current workflow, I never tell AI to just code X, I always approach it in such a way to leave space for research and breaking it up into multiple units of work (tickets) if the scope of the feature or idea calls for it.

I wouldn't say it's "interesting", rather that it's a natural result of incentive structures. If the AI labs try to sell their product as being good at finding bugs, then their expected valuation is the entire market cap of the existing static analysis industry, which is not very high. If they instead sell their product as being good at writing software, then their expected valuation is the "market cap" of the code-writing industry, a.k.a. the total spent on software development labor, which is very high. It doesn't matter what's true, only what promises to make them the most money.

This feels like just another variation of “it’s not the tool it’s the user,” which is too frequently leveraged during any discussion about AI. It makes it impossible to honestly assess the quality of the tool. You can always just say “people are doing it wrong.”

That problem is as old as commercial programming; it doesn't matter that now your actors are prompt engineer and LLM, the same story can be told with manager and programmer.

It is not in any way a “query thing” in the same manner as PageRank, it was based on prior work in deep learning and it is irrelevant that Deepmind was purchased by the same company that was behind Google Search.

I agree with you. It's gotten extremely good very recently and I think a lot of people haven't realised.

And they also won't believe you since they formed their opinion 6 months ago when code quality was 80% slop. Their loss though I guess? At least they'll be more surprised than us when they actually do lose their jobs? Maybe blissful ignorance is better!

Yeah it’s an interesting situation we’re in. I’ve seen people one shot really complicated tasks like “train a model to emulate DOS” or whatever. But I also believe if the Postgres maintainers fed every 600+ bug reports into Claude or Codex and shipped whatever came out the end result would be broken. I guess this is sort of like when machines learned to play chess - you’d expect a person who is really good at chess to be smart in other ways too, but the machines could only play chess. In the same way Codex can solve Millennium problems but can’t realize it shouldn’t hack into companies to improve its evaluation scores.

Then just have the same AI bug checks for new PRs and automatically reject any that fail.

Depends. Security is and never was only bug free software.

If no attacker reaches system X you are not as exposed as with your internet facing frontend.

Note how Robert mentions a sharp increase in back-patched commits. Many of these bugs have indeed been around, silently, for a long time.

> People (even the Rust fans) underestimate how many footguns are everywhere in C and C++.

The process of learning C++ (and to a much lesser extent, C) is almost exclusively one of learning about the footguns and how to avoid them. It's a life-long process, nobody ever finishes it.

I find it ridiculous how lean is treated like some magic pixi dust by people now.