Excerpt from a message I just posted in a #diaspora team internal f...
pod.geraspora.de
Excerpt from a message I just posted in a #diaspora team internal forum category. The context here is that I recently get pinged by slowness/load spikes on the diaspora* project web infrastructure (Discourse, Wiki, the project website, ...), and looking at the traffic logs makes me impressively angry. In the last 60 days, the diaspora* web assets received 11.3 million requests. That equals to 2.19 req/s - which honestly isn't that much. I mean, it's more than your average personal blog, but nothing that my infrastructure shouldn't be able to handle. However, here's what's grinding my fucking gears. Looking at the top user agent statistics, there are the leaders: 2.78 million requests - or 24.6% of all traffic - is coming from Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot). 1.69 million reuqests - 14.9% - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonb...
  • Soyweiser@awful.systemsEnglish
    3·
    5 days ago

    Re the blocking of fake useragents, what people could try is see if there are things older useagents do (or do wrong) which these do not. I heard of some companies doing that. (Long ago I also heard of somebody using that to catch mmo bots in a specific game. There was a packet that if the server send it to a legit client, the client crashed, a bot did not). I’d assume the specifics are treated as secret just because you don’t want the scrapers to find out.

    • YourNetworkIsHaunted@awful.systemsEnglish
      2·
      18 hours ago

      You could probably do something by getting into the weeds of browser updates, at least for web traffic. Like, if they’re showing themselves as an older version of chrome send a badly formatted cookie to crash it? Redirect to /%%30%30?

      • Soyweiser@awful.systemsEnglish
        1·
        17 hours ago

        Yes, there I heard there is some javascript that various older versions of chrome/firefox don’t properly execute for example. So you can use that to determine which version they are (as long as nobody shares that javascript with the public. So this might even not be javascript, I honestly know nothing about it just heard it).