Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn’t use postgres). From what we can tell, the attacker did not do anything with that access, so we don’t believe any user data was accessed, but we can’t be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it’s as smooth as we think it will be, but we can’t be sure!
As stated earlier, our writefreely instance is still up and running as it wasn’t impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don’t plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won’t be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn’t seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
As ever, both of your dedication to transparency and communication is both excellent and very gratefully received! I did see you said in the post above Ada that keypair rotation and all that that entails would make Piefed a tricker recovery but I was wondering if you had any updates for Piefed recovery? If you don’t, you don’t :) I fully appreciate how time consuming dealing with all this is, I just thought I’d ask.
We just ran out of time to get Piefed back yesterday (Australian time). We’re also navigating around moving house and the Kaity’s day job. It will be up as soon as we can today (It’s currently 7am here)
Yeah I thought the timezone thing might be playing a part! I hope I didn’t come across like I was complaining in any way as I definitely am not :)
Oof. People can be shitty. It sounds like not collecting data spared a lot of potential damage, though, so good on you for that. I know was the point, and this just proves why that was such a good decision.
Oh, minor thing: there’s some piece of punctuation in the name pgcrypto that your blog software is interpreting as markdown to start and stop italics. Luckily you mention it twice in fairly quick succession, so the passage in italics is relatively short, but you might want to escape out that character. I realize this is an unbelievably low priority right now, and possibly forever. It’s just something I would want pointed out to me if it were my post.
It’s good to be back
Had a feeling that this was the fact. Glad to be back
So happy you’re back! I was so desperate as to visit Reddit a few times. It was horrible. You’re amazing! ❤️❤️❤️
Thank you so much, Ada and Will. Appreciate the transparency! :3
To all curious, for the future: if you cannot go to your account on the Blåhaj instance or open up any stuff from there, check the desktop website of the instance, just go to lemmy.blahaj.zone (or its piefed equivalent).
Chances are, that there may be something on it. If you have an alt, I’d recommend one on an instance that’s mutually federated with the Blåhaj one.
For changing passwords, your app may not support it - use the desktop environment.
Consider donating to the Blåhaj instance! (Put the donation spot in the sidebar too!)
This feels like an inappropriate time & place to plug my instance, but I have a bot set up to mirror instance bans from blahaj (and dbzer0), so my instance can be a safe space as a back up for folks who need one. Important differences in that regard though are that I do have downvotes enabled, federate with hexbear, and could only copy bans going back a little over a year (which is over 3000 accounts banned btw! They do so much work to keep a safe space it’s jaw dropping- I donate monthly as a thanks for the ban list lol)
Are IP addresses stored? And if so are they affected?
It looks like IP addresses are stored in the DB in lemmy. It’s possible that the attacker had access to those IPs, but we don’t believe they accessed them.
This is the sort of thing we would turn off if we could :\
I would like to bring up my onion service post from earlier https://lem.lemmy.blahaj.zone/post/22655537
Neither side of the connection knows the ip address.
I don’t know anything about server management.
I believe IP addresses are anonymized on hexbear although I dont know how it’s done.
We could do that by direct DB manipulation.
I just use a VPN. Paid one, but not one that’s advertised everywhere.
thank you for all your hard work Kaity, Ada, and the rest of team, and for the transparency. even tho this was a horrible thing, the honesty and work makes me hopeful in a dark world. lots of love <333
Guess we now know where the database problems where coming from.
Thx for resurrecting us back.
The feeling when a small hobby non-profit project gets hacked and the owners quickly respond to the users and say “hey, we got hacked but don’t worry, your passwords are safe because they were encrypted!!”
But a damn multi-billion company gets hacked, takes months to tell the users and their answer is: “so… a few months ago, we got hacked, but it wasn’t that bad so we didn’t think about telling you until someone found our database for sale in a forum. Also, change your passwords, email, physical adress, bank account, credit cards and if you sent it to us, your SSN, because we didn’t think it was important so it was all stored in a plain wordpad file without any encryption”.
I know this must have been awful for you guys, but damn if it feels good to know that even if the fucker got access to your database, they couldn’t do shit because you were competent and took measures to protect your users in a way a multi-billion company doesn’t.
I get your sentiment, but the difference is the mega-company has to worry about what they say for when they inevitably get sued.
No one is going to sue blahaj, and their currency is trust and communication, so it helps to be open.
I’ve been on Lemmy for a while and the sheer naive idealism on here boggles me sometimes. You guys think “megacorp is evil and stoopid”, when realistically the person who discovered the breach freaks out because he may lose his job and is the sole breadwinner of the family, then takes it to his manager, who isn’t in a much better position. It bounces through a few low level admins and teams trying to figure out the extent, since there are hundreds of applications and systems, all while the clock is ticking by what you want to see. Only once they have things does it get bubbled up because they want to make sure they can answer to the VP in hopefully a way they won’t be scapegoated and lose their jobs. Finally it may get to the people who do have control of information but by now it’s been a while and past your idealistic timeframe. All because a common working man doesn’t want to become homeless.
But sure, blame the corporation, because it’s a single monolithic entity that you can focus your ire on instead of showing some sympathy for the working people, who are also here on Lemmy, fearing for their jobs.
he may lose his job and is the sole breadwinner of the family
Corporations are responsible for him being in that position. Hope that clears it up for you.
Uh, what entity do you think it’s responsible for the environment you just described?
No, the difference is that blahaj encrypts user passwords while a multi-billion dollar company stores them in a fucking plaintext file (alongside the credit card numbers and other sensitive data).
Also, under GDPR, a company must inform of a databreach ASAP, and they only do when they get caught.
See, this is why I respect the hell out of you Ada. Well, one of the reasons, because there are plenty more. But this is a perfect example of the kind of person you are, as well as the kind of admin. Transparency, rapid response, and you even opened up with an apology for someone else having screwed things up.
That goes for the entire blahaj team, but you are very much the face of it, and I just wanted to say something that I very often think, that we’re all damn lucky you’re here.
Fingers crossed this gets sorted out, blahaj.zone is such a blessing.
Thank you 🫂 I’ve been through some disastrous code deployments, but I know those experiences could never truly compare to something like this- stress, fear, accountability, and just feeling violated. You all must have put in sooo much effort and had to make some difficult decisions. Thank you for all of your time and knowledge to creating and supporting this space for us 🩷
Thank you for your hard work and transparency, Blahaj Zone team!
