• smeenz@lemmy.nz
    link
    fedilink
    English
    arrow-up
    4
    ·
    4 days ago

    If sites start blocking googlebot en masse, then googlebot will just start ignoring robots.txt

      • smeenz@lemmy.nz
        link
        fedilink
        English
        arrow-up
        1
        ·
        3 days ago

        Then the user agent string will just quietly become randomised so you can’t match it reliably because it turns out that honouring robots.txt was always little more than a “gentleman’s handshake”.

    • ℍ𝕂-𝟞𝟝@sopuli.xyz
      link
      fedilink
      English
      arrow-up
      4
      ·
      4 days ago

      Can they just put an EULA on the site and then sue Google for unauthorized access?

      Not in the US of course, but in the EU or something