Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
When one of these guys attacks your site, do they send the info back to the spoofed address or does the scraped info go to their real IP address? Is there some way to get a fix on the actual bot and not on some home user that got his network facing IP address hijacked?