Anubis, which block AI scrapers from scraping websites to death, has been downloaded almost 200,000 times.
  • medem@lemmy.wtf
    46·
    10 days ago

    Well, now that y’all put it that way, I think it was pretty naive from me to think that these companies, whose business model is basically theft, would honour a lousy robots.txt file…