Edit - We’re back!
We’ve had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.
As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it’s got plenty of space to spare, but it’s not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that’s happening, all of our services are running slow.
The good news is, we’ll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!
Both! When we were down, content didn’t federate to us. The servers trying to send that content to us will then retry and send it again, and each time it fails, they wait longer until they try again the next time. As a result, the stuff we missed will start rolling in at different times, depending on which instance it’s coming from.
But also, it’s normal behaviour for vote counts to differ, because we ignore downvotes, most other instances don’t. We have defederate some instances, which other instances may not have (or vice versa) and as a result, we will show different vote totals because we don’t have the same voting sources.