Ah yeah I need to refresh the data, ill do that later

Nobody reported it as down to me, I can bring it back up

Been working on some other projects recently so havent really looked at that site much

We just added a couple more people to have server access and ideally that situation wont happen again

That problem specifially could only be reproduced on the main server so fixes couldnt be tested without taking the entire site down and at some point after the 0.19.5 upgrade (when the issue happened) it became too late to roll the server back since there was too many activities that came in but if it happens again after an upgrade we would roll back pretty much instantly)

dbzer0 does seem like a better option for general privacy discussion though

lemmygrad isnt defederated from but many of their communities are hidden from the feeds by default (so you dont see them unless you subscribe to them)

same thing for some other instances like lemmynsfw

ones that are more recent wont be hidden since I havent been able to due to the issues but I should be doing another hiding round soon

Sites back. I reindexed every table although the only one that seemed to be affected by the issues was the comment table (it had duplicates I had to remove similar to what I did for the post and community tables before)

Site will slowly catch up in activities over the next couple hours so youll see posts, etc. that were made in the few hours it was down slowly come in

No votes or comments were connected to the duplicates but yeah if there was some you would move it over to the other one

The posts that had comments were the posts that were uniquely on the other community and those moved over with the post when I updated the community of the post

Sites back, communities that previously had their community page not vieweable are now viewable

An example of this is https://programming.dev/c/a11y

Basically what I did was before the maintenance exported every single post in the post table and use a js script to check for duplicate ap_ids (checking using sql itself didnt work due to broken indices so it was always missing some). Then I generated sql statements to remove duplicate posts that had higher ids than the other posts theyre a duplicate of. Ran all those during maintenance and since all duplicate ap_ids were removed I could now reindex the table to fix the broken indices

Did the same for communities after the posts (but also did some statements to move posts from one version of the community to the other)

Site might be slow for a tiny bit as it processes the 2 hour of activities that were generated while it was down

select them all, then ctrl+/

what frontend are you using, default lemmy one or one of the alt frontends .world has? Default frontend should handle it fine but havent checked the alts

versions that support (lower rectangle / higher number is more recent):

usage per version:

last major browser that didnt support which was safari updated to support it 4 years ago. update your browser

Sites back up

Got a lot of information about what it actually causing the issues which should help me automate some fixes for tomorrow + the weekend

There likely will be some more downtime tomorrow if I have time (if not itll be on saturday)

The issue is essentially a chain of issues with one needing to be fixed before I can fix the others. Some tables have duplicates entries which is preventing reindexing (since the indexes are broken). First one im going to try to fix is the post table.

I did some changes to remove a duplicate community to test which is https://programming.dev/c/photography@lemmy.world and ive managed to make it merged and now comments will show if you sort by comments in the community. Posts however still aren't showing up due to the posts table having the issue with duplicates as well. (fixing posts first so I can set all of the posts to the correct community in an automated way later for the communities)

around half the instances I checked were affected, no idea why only them were affected. world is also currently behind in inbound federation due to the pure amount of activities its generating atm (only instance that is atm). We fixed the issues so it shouldnt be happening anymore (and I can see this post on world)

world, lemm.ee dont have them, beehaw does for example

no problem, been using it as an excuse to learn about godot as well