Was anyone else unable to connect to slrpnk in the last week or so? Was the server down or was it something on my end?
Just saw the other post about the server outage so feel the below is relevant:
Is there a way to make slrpnk more resilient by having community members host mirrors of the instance? I also feel this is in line with the ethos of solarpunk where we’d rather have many decentralised instances that can support each other and provide mutual resilience, rather than a single critical instance.
What are people’s thoughts?
(I might be able to volunteer my home server for this if I can cover the requirements)
You must log in or # to comment.
Doesn’t the Forumverse already do this by default?
Lemm.ee was closed a long time ago, but all of its communities still kind of exist, because every instance that had federated with its communities, have full copies of those communities. For example:
- https://piefed.social/c/cartographyanarchy@lemm.ee
- https://lemmy.world/c/cartographyanarchy@lemm.ee
- https://slrpnk.net/c/cartographyanarchy@lemm.ee
Those three are the same community, but also not. Now that lemm.ee is not around to do the federation for that community, no data gets transmitted between the three versions of that community I linked above. Just look at the number of upvotes: They are different behind each link, because there’s nothing telling lemmy.world that someone on slrpnk.net had upvoted something there. You could even add new posts in any lemm.ee communities that are not locked! Those posts would only be visible for people on your own instance, though.
In the case of a permanently dead instance this is something of a problem, actually, because people might not understand that they are posting in a community that will never again federate.
But, for example in the case of the downtime of slrpnk.net, people were still able to use the communities. They were able to upvote and downvote, they were able to write comments, and they were able to make new posts.All that content was stuck in each individual instance, so if I from nord.pub wrote a comment there, a user on lemmy.world would not see it. But, when slrpnk came back up, all instances sent all of the updates to it, and it then federated then onwards to other instances, joining all the added comments into one.
The whole Forumverse functions through mirroring. Each instance has a full copy of any community its users have subscribed to. If the home instance of the community goes down, the copy will continue to function.
So, to write all this in the nutshell: While slrpnk.net was down, people using other instances were still able to use its communities without even noticing the server was down! Therefore, I would argue that what you are asking for, is already taking place. It does not work very well if slrpnk.net ever goes down for good, but as long as it eventually climbs back up after disappearing for a while, this solution works very well.
And also: If lemm.ee somehow ever comes back online, all the upvotes behind those three links will be federated to all other instances, and then any of those three links will show the same amount of upvotes.
All that content was stuck in each individual instance, so if I from nord.pub wrote a comment there, a user on lemmy.world would not see it.
We actually discovered right after the outage that Piefed instances appear to be able to see posts from other instances made to an offline or backlogged community. In particular, while viewing from piefed.social, I was able to see a lemmy.world user’s posts to !climate@slrpnk.net before slrpnk.net had caught up on the backlog from lemmy.world.
What you describe is certainly true of Lemmy instances, though.
Thanks for writing this out! It really helped me understand how this all works
When we first set up this instance, I thought about ways to do so, but it isn’t easy.
I still plan to add more resilience to the media hosting by sharing it as a Garage S3 service across multiple locations eventually, but for Lemmy or Piefed itself it would require quite a bit more effort than I am currently able to invest into this (and I am also not very knowledgeable about clustering a PostgreSQL database across multiple physical locations).
Happy to contribute my expertise as needed
Quick thoughts:
- It would expose user data to more admins
- Someone would still need to update DNS
- Data synchronization is not trivial
- it’s an additional overhead
Still like the general idea though
Yeah I agree - from the research I’ve done it doesn’t seem to be an established Fediverse practice so I couldn’t find any off the shelf tools or guides but I would be interested in teaming up with people and working on a novel solution if need be
I would be interested in finding others to share a https://garagehq.deuxfleurs.fr/ S3 media storage cluster. Would your server be somewhat close to Portugal (<200ms network latency) and have at least a terabyte free storage? I recently set up a local 4TB storage node with local replications, but the ultimate goal would be to add some remote hosts to it as well.
yeah I think I could spare 1TB of storage and my ping to Portugal is around 50ms
Good to know. I’ll keep that in mind and will message you once I find the time to work on this a bit more. Don’t expect it to be soon as my todo list is really swamped right now. But you could look into hosting Garage already on your server regardless on when I get around looking into it again.
cool - yeah I can have a look myself but let me know!
I add “resilience” by using an alternate lemmy or piefed login (i have one of each for precisely this reason)
Seems like a lot of ballache for the Admin in doing what you suggest ?
yeah I mean I do have an account on another instance but all the communities on this instance are still unavailable while the server is down - but yeah it would be more of a hassle (as any multi-region replication is)
