Creepy Crawlies (people.kernel.org)

  > because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.

I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.

Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.

Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:

  > Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.

  >  I really don't see why Safari on a modern iPhone should be so slow at this

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.

Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.

Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.

> Why is git.kernel.org “interesting” to crawlers

I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.

The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.

Tavis Ormandy called this, about Anubis, almost exactly a year ago:

https://news.ycombinator.com/item?id=44962529

It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.

  > meaningful dent

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.

It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.

Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.

I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.

The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.

Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.

It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.

I really enjoyed the writing style in this article.

And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.

Ironically, defense by obscurity may be the way to go here.

Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.

You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.

While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.

However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.

Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.

However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.

I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.

Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.

I still haven't found a way to deal with rotating residential IPs however, and most likely never will.

My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.

Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?

I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.

I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.

Just for fun, because I could, I vibed up a `cgit` replacement that runs entirely in the browser -- point it at a git repo where you've run `git update-server-info` and it'll load files as if it's starting to clone the repo, using range requests and browser caching to avoid actually loading more data than necessary for the view you've requested.

I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.

It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb

This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.

Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.

this is an increasingly common situation. it goes something like:

  - i have a free, niche resource
  - it becomes too popular
  - i make it more efficient
  - now it's really popular, and people are "abusing" it
  - let's make them proof-of-work
  - ... and proof-of-work harder
  - but now "legitimate" users can't use it
  - ???

the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?

which is called l402/x402. micropayments' day has come

Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.

It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.

> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.

As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”

I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?

Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)

> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.

If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.

> Why is git.kernel.org “interesting” to crawlers

Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.

High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?

> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.

How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally.

If someone else wants to run a public HTML service, let them deal with the bots.

If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.

Sucks, but here we are.

I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).

Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.

There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are.

Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.

FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.

Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.

from what I see there are 2 solutions:

1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally

2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.

If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.

CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.

There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.

it's getting insane, I have a high profile client, I manage their infrastructure including web server. I swore to them years ago they would not have to turn on the CF managed challenge / under attack / human verification.

I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.

and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.