LLM bots absolutely don’t care about any stated preferences. Heck, they don’t care about robots.txt rules, which try to give hints to search engines what’s even sensible to add to the indexes.
However, “this isn’t our fault, we told them this shouldn’t be scraped, but did this dumb bot listen, well no” is still good policy.
LLM bots absolutely don’t care about any stated preferences. Heck, they don’t care about robots.txt rules, which try to give hints to search engines what’s even sensible to add to the indexes.
However, “this isn’t our fault, we told them this shouldn’t be scraped, but did this dumb bot listen, well no” is still good policy.