this post was submitted on 22 Sep 2024
18 points (100.0% liked)

Fedia Discussions

17 readers
1 users here now

founded 2 years ago
MODERATORS
 

Hi all. Several of you have reported problems with fedia.io not federating with other instances correctly.

The cause is that rabbitmq crashed, but not all the way. It crashed to the point where new connections would timeout, but the service was still running such that it wouldn't auto restart. I will be creating some automation to detect that proactively and restart rabbitmq if/when it happens again.

top 21 comments
sorted by: hot top controversial new old
[–] hitstun@fedia.io 5 points 10 months ago (1 children)

Thank you! A thread I just posted made it to Lemmy instances within 15 minutes. The thread that didn't federate two days ago has found its way there too. Jerry is the best instance admin and Fedia.io is worth paying money for.

[–] jerry@fedia.io 5 points 10 months ago

Once it is more reliable, then I’ll agree, but thank you

[–] Chozo@fedia.io 3 points 10 months ago (2 children)

Are we still having trouble? A thread I posted 45 minutes ago hasn't shown up on the instance I posted it to yet. The "Open original URL" menu on the post goes to the Fedia page instead of LemmyWorld. It seems like a comment I posted 2 hours ago is also not appearing on the original instance, either.

[–] jerry@fedia.io 5 points 10 months ago (1 children)

ok - rabbitmq started having prroblems with the delivery queue again. I got it going again. Those messages should be delivered soon.

[–] dhhyfddehhfyy4673@fedia.io 2 points 10 months ago (2 children)

Seems to be busted again already.

[–] jerry@fedia.io 2 points 10 months ago

Working on it

[–] jerry@fedia.io 2 points 10 months ago (2 children)

ok - the queues are processing again. I will work on a more permanent fix after dinner

[–] Chozo@fedia.io 2 points 10 months ago (1 children)

Sorry to nag, but it looks like it might be acting up again lol. Doesn't look like any sync has happened for 5 hours.

[–] jerry@fedia.io 3 points 10 months ago (1 children)
[–] magnetosphere@fedia.io 2 points 10 months ago (1 children)

Thank you for all your hard work. I am experiencing the same problem, and was just browsing Fedia Discussions to see if anyone else had reported it. Again, thanks!

[–] jerry@fedia.io 3 points 10 months ago (1 children)

It’s almost caught up. My apologies. I am trying to get it fixed permanently.

[–] Chozo@fedia.io 2 points 10 months ago (2 children)

It's appreciated! <3

While you're working on a more permanent fix, is there a preferred way we should let you know about hiccups like this? Or is pinging you in a thread in this magazine sufficient?

[–] jerry@fedia.io 3 points 10 months ago

Note that even once I get this fixed, there will inevitably be another problem crop up. Posting here is fine, but an email to jerry@infosec.exchange or a ping to @jerry@infosec.exchange (my mastodon account) would probably be faster (note, when federation breaks here, messages to @jerry@infosec.exchange wouldn't get through from fedia.io...

[–] jerry@fedia.io 2 points 10 months ago

be aware that this is fixed now

[–] dhhyfddehhfyy4673@fedia.io 1 points 10 months ago (1 children)

Cool, thanks for the prompt work.

[–] jerry@fedia.io 2 points 10 months ago

be aware that this is fixed now

[–] jerry@fedia.io 3 points 10 months ago

I will check.

[–] testing@fedia.io 1 points 10 months ago (1 children)

any guess why visibility on lemmy has been heavily affected, whereas from microfedi things looked normal?

[–] jerry@fedia.io 5 points 10 months ago (1 children)

Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.

The queue appears like it’ll take several hours to flush, but it’s working.

[–] testing@fedia.io 2 points 10 months ago (1 children)
[–] jerry@fedia.io 1 points 10 months ago

be aware that this is fixed now