You might remember a company called Google. In 1998, they came up with an amazing product — a web search that didn’t suck! And it could find things! With the state of search in 1998, this was incre…
  • smeenz@lemmy.nzEnglish
    1·
    3 days ago

    Then the user agent string will just quietly become randomised so you can’t match it reliably because it turns out that honouring robots.txt was always little more than a “gentleman’s handshake”.