For the context: https://feddit.org/post/2374543

TL;DR: Aussie.zone has been experiencing a 7-days delay with LW since quite a long time due to the way Lemmy manages federation, and the distance between LW and aussie.zone.

Lemmy.world is the only one having such delay, even Lemm.ee which is the second most active instance is up-to-date (same for Lemmy.ml with a lot of active communities)

Graphs:

The issue has been solved in 0.19.6 (https://github.com/LemmyNet/lemmy/pull/4623) which hasn’t been released yet.

Explanation by @Nothing4You@programming.dev

lemmy’s current federation implementation works with a sending queue, so it stores a list of activities to be sent in its database. there is a worker running for each linked instance checking if an activity should be sent to that instance, and if it should, then send it. due to how this is currently implemented, this is always only sending a single activity at a time, waiting for this activity to be successfully sent (or rejected), then sending the next one.

an activity is any federation message when an instance informs another instance about something happening. this includes posts, comments, votes, reports, private messages, moderation actions, and a few others.

let’s assume an activity is generated on lemmy.world every second. now every second this worker will send this activity from helsinki to sydney and wait for the response, then wait for the next activity to be available. to simplify things, i’ll skip processing time in this example and just work with raw latency, based on the number you provided. now lemmy.world has to send an activity to sydney. this takes approximately 160ms. aussie.zone immediately responds, which takes 160ms for the response to get back to helsinki. in sum this means the entire process took 320ms. as long as only one activity is generated per second, this is easy to keep up with. still assuming there is no other time needed for any processing, this means about 3.125 activities can be transmitted from lemmy.world to aussie.zone on average.

the real activity generation rate on lemmy.world is quite a bit higher than 3.125 activities per second, and in reality there are also other things that take up some time during this process. over the last 7 days, lemmy.world had an average activity generation rate of about 5.45 activities per second. it is important to note here that not all activities generated on an instance will be sent to all other linked instance, so this isn’t a reliable number of how many activities are actually supposed to be sent to aussie.zone every second, rather an upper limit. for example, for content in a community, lemmy will only send these activities to other instances that have at least one subscriber on the remote instance. although only a fraction of the activities, private messages are another example of an activity that is only sent to a single linked instance.

to answer the original question: the week of delay is simply built up over time, as the amount of lag just keeps growing.

  • onlinepersona@programming.devEnglish
    3·
    2 months ago

    It might be worth proposing as an addition. I think we all know that batching is better for the network and processing. Parallelization might work now, but the network grows (and it probably will), large servers might have more than ~6 activities/second to send.

    Anti Commercial-AI license

    • db0@lemmy.dbzer0.comEnglish
      4·
      2 months ago

      I think if they make the amount of activities you can send adjustable (automatically or manually) it can also scale. The problem is that it will also cause a lot more load, but I’m not quite certain by how much. But I will agree that apub needs to extend itself to allow batching federation activities as it’s bound to become an issue eventually.