The soluition here is to use any other instance then the 3 big ones: lemmy.ml, lemmy.world and bewhaw.org Lemmy instances scale horribly, the more users the worse the performance.

See this piechart from a week ago, the top 3 have over 50% of the users: