Benchmarking Bot Detection Systems Against Modern AI Agents in 2026 (research.poh.org)

49 pointsby mdahardy5 days ago8 comments

vikrantrathore 4 days ago

In your bot detection systems I do not see open source system like anubis, safeline or sibuna (I am the author of sibuna). Sibuna is built as self-hosted alternative to cloudflare with console UI providing OWASP CRS, JA4 TLS signature from ingress, these are still necessary besides bot detection and cloudflare offer it as enterprise offering.

I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.

conception 5 days ago

Based on trying to buy anything retail that is scarce, bot detection systems are definitely not working. I wonder how their system would do against https://refractbot.com/ or other popular retail bots.

routerl 4 days ago

To be fair, most bot detection can't and isn't trying to catch 100% of bots. Generally the idea is to prevent repeated abuse, rather than stop every single instance of it.

In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.

aurareturn 4 days ago

When they run these benchmarks, they should measure false positives - because it's annoying for a real human to solve a captcha.

If blocking 1 bot request is +1 then making a human solve a captcha should be something like -100. That's how we should grade.

nickabe 4 days ago

I feel like newer systems are even worse, probably in about 50% of my online experiences now I just get "banned" immediately. Opening a new account = sign of a bot = banned.

Hyper focusing on detecting bots without thinking about the humans you're blocking is making the Internet a crappier place.

asksomeoneelse 4 days ago

I don't want to say "that's an ad", but I can see an obvious conflict of interest there.

Also, the methodology just compare the true-negative detection rate for known negative, it seems?

> "Proof of Human identified 99% of automated sessions"

> "We measured detection rates for automated sessions but did not evaluate false-positive rates among human respondents."

Trivially easy measure to game if so. "isHuman(x) = return false" gets you a 100%.

AznHisoka 4 days ago

Did they use the browsers against sites they owned or 3rd party ones? If the latter, then lots of sites have Cloudflare Bot Management on, without actual strict restrictions on bots (a lot of sites using CLoudflare as a CDN for instance)

mimimile 4 days ago

While building a project, I needed a crawler to read robots.txt. Out of 10,000 sites, just over 1,200 refused access outright.