Bot escalation

The neverending battle continues.

A couple of months ago or so I had landed on a potential mitigating tactic to use against the onslaught of so-called "AI" bots monopolizing the bandwidth here aboard StarshipTim.com. It worked for a while, but those bastards, just like the Borg, adapted, and now the problem is worse than ever.

I am at my wits' end. These (presumably) scraper bots have managed to circumvent all barriers against them by convincingly camouflaging themselves as human users and rarely/never using the same IP address twice. Blocking them has become impractical without taking drastic measures.

In and of themselves, these bot visits aren't debilitating. They are most likely illegal in the sense that the scrapers scouring the Internet to feed large-language-model algorithms don't give a tinker's damn about copyright law, but in practical terms, what's to be done about that? At some point there may be a class action suit or several to take part in, but proving theft is tricky and the operators of these bots know it. But each visit isn't a drain on resources by itself. Cumulatively, however...

Since the calendar turned to 2026, there have been 203 visits to StarshipTim.com, not counting hits from myself when posting and checking comments. 185 of them are from unwelcome bots. Nearly 2% of the bandwidth this account is allotted for the month has been used already by bots. If that usage rate stays consistent, 12-15% of my bandwidth for the month will have been stolen by bots circumventing the "keep out" signs. This is a rather low-trafficked site, so that won't hurt me here, but really there's no reason to think the bots will keep themselves to this pace.

I host other sites on this server too, and the larger/more-trafficked of those are hit much harder. On one such site just one percent of visits today were (probably) human users, the other 99% were unwanted bots stealing bandwidth (and stealing content for their LLMs). Gigabytes of bandwidth have been used on that account since 12:00:01am January 1st, almost all of it bot traffic. Typically, that site will use about 30GB per month; last month it used 110GB (mostly bots), this month it's already on pace to use about 150GB, far exceeding its allowed usage. I've been borrowing bandwidth from other accounts to cover the excess so that one doesn't go offline for violations. The client isn't at fault. Elon Musk and his ilk with their scraperbots are.

Unsurprisingly, there is an industry starting to sprout around fighting the bots. This abuse of the system has given rise to a profit opportunity for companies that will basically envelope a site into its own bubble and charge thousands of dollars a month to mitigate the bot traffic. Useful, I guess, for giant corporate websites where adding a several-thousand-dollar-per-month expense beats the loss of resource to bots expense. Cloudfare at least offers something more affordable for the little guys, $20 and $200 per month services that attempt to filter bots by routing your visitors through their server security first, then sending those that clear to your own server. I wonder about the false-positive rate with such a thing.

This is maddening because it's a new example of our societal profiteering on bad behavior. Take a walk around the neighborhood, see how many homes have ADT or some other security service's sign in the yard/window/whatever. These people pay every month for protection against bad behavior. Nice home you got there; be a shame of something happened to it. Now imagine that a bunch of Techbros were running a systematic campaign of stealth home invasions that copied and photographed all of your possessions while running up your electricity and water bills, and that these invasions happened every day, usually more than once. Now further imagine that law-enforcement didn't care about these invasions, that lawmakers were browbeaten into thinking that these invasions were somehow permissible because they involved a digital computer program that allows for entry into the home, so technically it isn't breaking-and-entering, and none of your stuff is actually missing when you get home, so is it really theft? With all that going on ADT would be set to make a fortune in new subscribers, right? Symbiosis.

I'm actually experimenting with one of these pay services, the smallest scale option, to see if it makes a lick of difference. If it does, then I can offer it to clients as a mitigator, but I don't like it. It feels like letting the terrorists win.