this post was submitted on 23 Jun 2024
345 points (98.1% liked)
Technology
57448 readers
4619 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Novice web site owner/coder here: wondering if I can block them somehow via IP address in addition to robots.txt. Server firewall rule? Remember, I said I was a novice....
You can block an IP but first you would need to know which IPs are scrapers. And they could just use a VPN to bypass IP blocks.
Yes, the less expensive VPNs especially have a lot of users using the same IP addresses.
You can get a VPN with private IP's but this is more expensive. For a company of OpenAI's size that would be a drop in the bucket though.
Google "spider trap website" or something.