‘Pervert’, much like the word ‘deviant’ means someone engaging in behaviour that deviates from the norm, usually in what is considered to be a morally unacceptable way: you can see this in phrases like “perverting the course of justice” or “perverse verdict” which aren’t to do with sex. However, in common use, the word pervert was so closely associated with sex crimes due to its scandalous nature that most people tend to use it on its own as referring exclusively to paraphilic acts and the people who engage in them.