TechTakes

I've kinda said this before, but I expect shit like this to feed a growing trend of treating all crawlers as malicious until proven otherwise, and blocking them accordingly - even if it causes serious problems, its arguably better than the risk of getting fucked by an AI crawler.

I also expect robots.txt to fall out of use as a consequence of this plague of AI crawlers. Between OpenAI openly ignoring it, Perplexity openly deceiving it, and Anthropic spamming crawlers to get past it, the trust necessary for robots' "voluntary compliance" model to work has been thoroughly fucking shredded.