Edit: obligatory explanation (thanks mods for squaring me away)…
What you see via the UI isn’t “all that exists”. Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see “under the hood”. Any instance admin, proper or rogue, gets a ton of information that users won’t normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.
Edit: To clarify, not just YOUR instance admin gets this info. This is ANY instance admin across the Fediverse.
Except if you live in Saudi and you voted in support of gay rights. Or HK and voted something negative of China, or if you were a republican and voted something supportive of trans. Voting behaviour can be published to bring detriment to their users.
In the UK, the Labour party suspended members who voted for content from other parties (such as that they had recovered from covid) etc or for voting for content from groups that were proscribed later and they were excluded from the party for this sort of stuff. They had previously hired someone who previously worked for a company linked to the Israeli state and potentially Mossad and they were using this knowledge to crack down on left wingers with in the party. (https://www.theguardian.com/politics/2021/mar/02/lawyers-complain-to-labour-over-hiring-of-ex-israeli-intelligence-officer)
What people upvote is increasingly being highlighted and used as discreditation by association.
As the world becomes more digitalised, and this is used more and more, it becomes more and more critical. Maybe not for you personally, but for many.
That account still has to be linked to a real person, and not doxing yourself circles back to using social media responsibly. Social media is just that, Social media. You cant swim in the pool and not get wet. The ways in which your describing it being abused are ways in which every facet of every social media can be abused. So if one is so concerned about it, why even bother using social media.