AI crawlers problems (and solutions)

[boreal_cryptid]

recently AI crawlers have been even more aggressive. i decided to start a topic about it, and about possible solutions.

stories so far:

• Open source devs say AI crawlers dominate traffic, forcing blocks on entire countries
• Amazon's AI crawler is making my git server unstable

possible solutions:

• Anubis - Anubis is a Web AI Firewall Utility that weighs the soul of your connection using one or more challenges in order to protect upstream resources from scraper bots.
• ai.robots.txt - A list of AI agents and robots to block. There's limitations to this. The most obvious one is that many bots will simply not respect or honor your robots.txt. The ones from major companies supposedly do. Think of it as more of a polite request, rather than a "block".

[crazyroostereye]

This is difficult subject, as you have Anubis which is a bit of a Nuclear Response to Scrapper Bots. As they also block good bots like Archivers and Web Indexers.

Where it ai.robots.txt is good list Scraper User-Agents. But it relays on the User-Agent which can be easily manuplated. And I heard Stories that the big Scrappers (Amazon and co) use typical Browser User Agents and ISP IPs for their Scrappers instead of their bot IPs. Tho I cannot confirm these Stories.

But I an say both Projects are Good Solutions, It's just sad that we have to use them.

[TheFrugalGamer]

There's also Glaze, which claims to scramble your images making them unusable by AI:

https://glaze.cs.uchicago.edu/

I have no idea how affective it is, but it's kind of hard to prove a negative, so it may just be that we have to wait around and see if any AI bots manage to scrap glazed artwork.

[Julikins]

I would second Glaze and Nightshade as well, but sometimes the issue comes with not having the sufficient amount of horsepower to get the program to run. And I understand a lot of folks are willing to help others and get the glazing applied on their end. I could also suggest something like Artshield as well. Was going to put Sanative here as well, but lately their login link isn't working so I opted to not add that here.

I want to say another way to go about it is have a light layer of an image that could confuse the AI crawlers but I can't say for sure what specifically but I remember bits and pieces of it (which is what's already been noted)...

[brisray]

I run my own web server and this morning (1st July) I ran my monthly log rotation and saw that June's logs were 1.70Gb in size (7,245,677 requests, 4,385,798 pages) rather than the usual 300Mb (1,400,000 requests, 1,000,000 pages).

I used Log Parser to list the user agents and here's some of the top rquesters:

GPTBot - 3,807,710 requests
Scrapy - 2,593,094
Barkrowler - 52,715
Meta-ExternalAgent - 42,502
ClaudeBot - 29,145
PetalBot - 20,653
MJ12bot - 14,399
Bytespider - 12,866

What to do about them?

There's loads of others such as the search engine and archiving bots but I am not worried about those. What does bother me are the AI bots and those where you have to pay to get their insights into the scrapes - basically, they charge users for data they got for free.

Most say they respect robots.txt. If that doesn't work then as I run my own Apache server I could use SetEnvIfNoCase or a rewrite rule to deny them access.

There are lists of bad bots but adding them all appears to me to be overkill.

[crazyroostereye]

I run my own web server and this morning (1st July) I ran my monthly log rotation and saw that June's logs were 1.70Gb in size (7,245,677 requests, 4,385,798 pages) rather than the usual 300Mb (1,400,000 requests, 1,000,000 pages).

I used Log Parser to list the user agents and here's some of the top rquesters:

GPTBot - 3,807,710 requests
Scrapy - 2,593,094
Barkrowler - 52,715
Meta-ExternalAgent - 42,502
ClaudeBot - 29,145
PetalBot - 20,653
MJ12bot - 14,399
Bytespider - 12,866

What to do about them?

There's loads of others such as the search engine and archiving bots but I am not worried about those. What does bother me are the AI bots and those where you have to pay to get their insights into the scrapes - basically, they charge users for data they got for free.

Most say they respect robots.txt. If that doesn't work then as I run my own Apache server I could use SetEnvIfNoCase or a rewrite rule to deny them access.

There are lists of bad bots but adding them all appears to me to be overkill.

Yeah I noticed similar to, but what I can recommend ai.robots.txt, they include the worse offenders AI scrappers. As those have the tendency to come back more frequent and call more processing Intensive calls, if you run a non Static site. Implementing them is also fairly easy as they already have preperred files for the most common Webservers. Not Perfect but it's the least intrussive I know.

[boreal_cryptid]

   new stuff: Cloudflare blocks AI crawlers by default and launches Pay Per Crawl for publishers

Cloudflare has rolled out a default policy to block known artificial intelligence web crawlers, aiming to prevent the unapproved collection and use of website content by AI companies. Under the new approach, domain owners setting up a site on Cloudflare are prompted to specify whether or not to permit AI crawler access, giving users immediate control over data scraping activities.

[Loebas]

I have all known AI bots denied in my robots.txt. But in my eyes robots.txt is just passive dont disturb sign, since it is up to the bot itself to honor it, or not. And experience learned that some AI bots are not like that.

Other option is to take the problematic ASN number, and convert it to a IP range, and deny those IP's access with apache's htaccess. Apache will just serve up a 403 to these bots.

If you own your own webserver, you can also use its firewall to block those bota aswell. Which is more effective, and saves more bandwidth.

In my eyes, you cannot just take someones hardwork to be used in a random LLM. Especially when the webmaster sees nothing in return. Its just stealing.