QGIS enacts moratorium on AI contributions (github.com)

so i made a simple ab testing eval on fixed prompts: https://threejseval.com

There is a 7 degree o freedom robot arm that is really telling of the improvements in coordinates reasoning:

https://threejseval.com/gallery?eval=robot-arm

I’ve been renovating a spatial/topology validation system for an OpenLayers driven map editor for robot fleets. I set up a clean interface for spatial indexes and predicates as well as your basic geotoolbox functions.

I was most surprised by a moment where Opus detected that using a union approach to ensure that robot footprints always sat within valid polygons (possibly straddling multiple valid polygons) was slow in testing. So it decided to find the difference between footprint and each polygon within the AABB of it. And if the remaining polygon is zero size, it’s valid. If not, the remaining shape is what is invalid.

I was also surprised just how good it was at recognizing, “in the test data there’s a lot of this kind of feature so we should explore what’s more common: hits or misses?

Before robotics I spent a lot of time building spatial models for data processing and analysis. I find that I can usually just explain the spatial question in natural language and it will figure out the relevant inputs from the dataset and what the process can be. I’m not obsolete quite yet though because maybe 10-20% of the time it goes on a pretty fantastical journey through unnecessary intermediate data products.

I think this is going to be a common outcome from the fact that many people have AI agents that can one-shot fix and deploy. Maybe it’s not a bad thing for those people and the maintainers who each get what they want.

The other option they proposed was a proof-of-humanity by payment, which seems much more in line with the above.

The problem with just banning AI is you didn’t solve trust, and your untrusted users are just going to drop drive-by PRs anyway. How do you know if an untrusted user did or did not use AI? In practice this likely means just aggressively closing “slop-looking” PRs from users you don’t recognize?

But then, you’re preventing your trusted community members from benefiting from AI assistance.

I’ve been advocating for defining a higher standard for AI contributors, eg an AGENTS.md that specifies some meaningful correctness bar. In this case, perhaps the PR that adds you to CONTRIBUTORS.md could to add a large novel and non-trivial end-to-end test; if you can construct problems that are easy to verify but hard to get right, then this is your “proof of work” to establish competence.

From reading the GH issue it does seem this is a placeholder solution, so I hope they find a more stable long-term resolution.

I have a sneaking suspicion that people are having some agent check out 500 repos and just prompting with “find a bug in this repo and fix it” and submitting the results blindly.

Edit: typos, on mobile

Most bugs that are reported via AI-generated GitHub issues are edge cases, which are technically bugs, but not something anyone actually ran into.

Contributions are exclusively AI-generated, and it's much easier to just go off of a GitHub issue myself than asking the bot behind the GH account to remove the 5 redundant tests it added, or to refactor what has become a 3k line file.

AI doesn't care about code quality, maintenability, or long term goals. It's just a local maximum optimized with RL. Great for debugging, OK for reviews, but can't produce anything that's above average.

https://www.youtube.com/watch?v=NnV_cWeoo5Q

One thing I've been thinking is something like the DCO checks. Set up some account bridge between (eg.) Discord and GitHub, then auto close anything submitted by accounts not sufficiently involved with a message linking to the new contributor policy and rationale.

Perfectly and succinctly said.

Most often we’re satisfied with signals and indicators that it works and we build theories and mental models that use to try to avoid the effort of proving anything.

Worse is what happens when you start using LLM models to generate the code for you. I find you start losing the mental model short-cuts; you start to trick yourself into understanding the code. You understand little.

And what few studies there are of code review suggest that we have very little impact on code quality after reading more than a couple hundred lines of code over an hour.

So the current mental leap is, “the AI is a better programmer than I am.”

I think it’s still worth becoming a better programmer regardless. And there’s no royal road. You just have to write code and struggle with writing bad code.

The bottleneck has never been the writing of the code. It's all the rest that takes the majority of the time. Of course, it is often the writing which lets you expand your understanding or your awareness of challenges and pitfalls, which then puts you back on the drawing board.

That’s the old “Programming as Theory Building” stated by Peter Naur. Writing code, once you learned a language and a few libraries has always been easy and can be made easier by using tooling. Having a good understanding of what has been written and what needs to be added is the difficult part.

I think for every line of code I’ve written, I spent 2 to 5 hours reading specs and docs. Creating software is always about understanding. If you’re thinking too much about syntax and libraries symbol, you’re just at the beginner phase of coding. Like thinking about grammar when learning a second language or being hyperaware of the car as a beginner driver.

One approach that Ruby took for type introduction was to put that in a different file. That comes to my mind.

Typescript brings some niceties, but also some boilerplate all other the place. Combined with something like Effect, Zod, and the like and line count explode. Not sure OCAML or Haskell would provide more compactness for equivalent safety.

My prediction is that in the future we are going to be more focus in the testing part (that should be more human readable) than the code itself.

Whatever it is, the only thing clear to me is that the tooling will have to change.

The way I like explaining this to people is that I don't think I build products anymore. Now I build the harnesses which build the product. And a massive part of that is ensuring that I can make the AI remain aligned with my way of thinking.

Is the fear that it is (maybe wilfully) misinterpreting the code, being too general, not going into specifics or is just not smart enough to catch all the nuances, privacy issues and hidden mistakes? Is it a token issue? A combination of all this?

I can’t help drawing a parallel with what is going on now. Even though I’m pro AI, I understand why QGIS is stepping back. There are a lot of places where the processes and tools are not ready yet to fully embrace it and you need time to prepare.

> AI stream is a lot higher than what human review can take seriously, leading to either a longer list of un-reviewed PR's or not fully reviewed code and quality issues. Both thing we don't want in the project.

It’s not about detecting AI, it’s about reducing the amount of code to review.

They don’t need to know. The policy is not a linter that will scan each PR. Just like the law does not monitor everyone 24/7 for criminal behavior. It’s enough that the rule is in place and it’s enforced when they catch an infraction. No need to be perfect as long as it lowers the pain.

Also, Linus is also comfortable closing 100 PRs as "this looks like garbage". Many communities are softer and try to be more helpful with new contributors.

The problem isn't just the code - the documentation for the change explaining what it fixes, how, why, what testing has been done, and so forth is often either lacking, far to verbose, or both (massively verbose without actually containing much useful information or other sense).

You raise a valid point for the future: when the average quality of submissions is good enough that it is impossible to tell the difference between human (or human aided by AI but giving its output sufficient review/rewrite) and LLM/agent submissions, then the issue will have gone away. But, given the problems projects like this are seeing it is obvious that the technology (or the people using it, probably both) are not nearly there yet.

> The best LLM policy…

That is a great policy. If only the entire world would follow it, or in fact even ever read it.

If the only reliable way for now to not be suffocated under a mass of bad AI submissions is to refuse anything that even vaguely sniffs of AI, then that is what many projects are going to have to do. If that offends you, then fork the project and patch your changes in, or vide code one from scratch yourself, then you can keep maintaining that along with all the submissions you get for fixes. Let time tell which the right decision is :-)

[1]: https://github.com/storytold/lightcraft

The fact of being open source means if your contributions are rejected, you simply need to fork. The open market competition will soon determine if your contributions were truly valuable and whether the original project's owner misjudged, or that contribution was essentially worthless.

Another option would be to make a contributor pay for the review of its PR, even a small fee, to deter AI-slop, at least for new contributors. It could/would exclude students or people with the lowest wage though, but I'm optimistic even a 1$ bill or less would be enough.

Traditionally you've opened to a PR only to get closed for a style/procedural issue or get ghosted. You often didn't fork because maintenance is a hassle. Less so with LLMs, you know?

Nowadays you can just implement your change and vibe-fork it...

If it does support projecting differing CRSs though, please can you point me at it? I’m a nobody and nothing to do with qgis but I would love to have a play with it!

Maybe it becomes normal to fork a project and vibe-code the extra thing you want and everybody does that and it's fine?

Doesn't seem like a loss for the project, just a win for you. If I understood the announcement correctly, they're doing this more for logistical reasons than to outright ban all AI-assisted code, so if something turns out to be super awesome then it's not like they have to do a clean-room reimplementation.

Meanwhile, let a thousand forks bloom, why not?

> My code is clearly AI "assisted"

No such thing, it's 100% your code, regardless of how it came to be.

The most egregious examples are always lazy (big diff, lengthy descriptions, weird phrasing,…). Now there’s a section in the policy to point to for the quick rejection.

This is a higher-order rephrasing of the "intelligence is compression" argument.

Life has gotten to the point where completely non-technical people have gawked at me for still having a job related to software. In their minds, AI can it all, and in their minds coding is already obsolete. The sample size for these opinions will only grow and the ratio of people that rightfully apply skepticism will drop, either from fatigue or burn out or by virtue of more and more people adopting the "AI is roughly AGI" crowd. For me a good metaphor is the concept of the selfie. Everyone mocked selfies at first, and now you'd be deemed insane to negatively opine about selfie behaviour. The same will apply to AI slop grenades (even of high quality output) in due time. I think it's inevitable.

Endless hoardes of people can now unleash never-ending arguments and suggestions and it feels like it will become impossible to defend against them because one minute only contains 60 seconds of time that you can spend refuting a false or <100% correct suggestion.

anthropic at least gives 6 months claude max pro to oss maintainers, but to me that seems like a drop in the ocean compared to the negative impact their product has on these peoples' lives.

> If we subsequently discover that you have used an AI, LLM, or any agent-based workflow for your contribution, you will be subject to code reversion and contributor rights sanctioning.

Policies like this have an element of trust to them. That's not new: open source projects have needed to trust contributors not to paste in proprietary licensed code since open source began.