• Throwaway@lemm.ee
      link
      fedilink
      English
      arrow-up
      10
      ·
      1 year ago

      You’d think so, but it’s just not. Pretend “Gamer” is a slur. I can type it “G A M E R”, I can type it “GAm3r”, I can type it “GMR”, I can mix and match. It’s a never ending battle.

      • Echo Dot@feddit.uk
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        That’s because regular expressions are a terrible way to try and solve the problem. You don’t do exact tracking matching you do probabilistic pattern matching and then if the probability of something exceeds a certain preset value then you block it then you alter the probability threshold on the frequency of the comment coming up in your data set. Then it’s just a matter of massaging your probability values.