this post was submitted on 01 Aug 2024
550 points (97.7% liked)
Technology
58009 readers
2949 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Robots.txt does literally nothing. It's a piece of courtesy that's easily ignored if you don't care.
Yeah but it stops bing and a bunch of AI scrapers that want to act like they're following the rules
How do we know it stops bing? As far as anyone knows they could have instructed their programmers to alter the crawlers so that it ignores robots.txt when on Reddit - that should have taken them a whole 2 minutes.
Reddit blocking any search crawler via robots.txt is such a non thing that it shouldn't even be reported.
Ok but they do respect it, we know that https://searchengineland.com/microsoft-confirms-reddit-blocked-bing-search-444385
They even have a page telling you how to use it https://blogs.bing.com/webmaster/June-2008/Robots-Exclusion-Protocol-joining-together-to-pro/