jerry

joined 2 years ago
MODERATOR OF
[–] jerry@fedia.io 2 points 8 months ago (1 children)

I blocked those 3 after getting so many reports that my hair was falling out. If y'all want them again, I can unblock them, but please please please stop reporting them, then.

[–] jerry@fedia.io 37 points 8 months ago (1 children)

A tariff can be helpful if, for example, the Chinese government decides it wants to dominate the world market in widget production and so subsidizes the production of widgets by Chinese counties. This has happened in the past with steel.

But the downside is so much worse. And we experienced it not that long ago. Good read is here: https://www.ers.usda.gov/amber-waves/2022/march/retaliatory-tariffs-reduced-u-s-states-exports-of-agricultural-commodities/

Basically, the US applied tariffs on a bunch of stuff from a bunch of counties. The stuff we bought produced in those counties or made from things produced in those countries became more expensive. (I remember washing machines becoming substantially more expensive as a result of the tariff since I had to buy one at the time).

But that’s not the end. Those countries applied retaliatory tariffs to stuff the US exports - mainly agricultural goods. Those things are commodities produced by many countries, so a bunch of them simply stopped buying the tariffed US products, and instead started sourcing them from places like Brazil. I’ll leave it an exercise to the reader to figure out whether there’s a link between US tariffs the accelerated destruction of the Amazon rainforest to satisfy sudden new demand for produce.

And here we are years later and while many of those tariffs are gone, the US agricultural industry never recovered much of that lost business.

[–] jerry@fedia.io 2 points 8 months ago

Apologies for just seeing this. I will take a look. Mbin has some issues.

[–] jerry@fedia.io 1 points 9 months ago
[–] jerry@fedia.io 4 points 9 months ago (3 children)

Apologies - I've been having all manner of problems with mbin processing inbound federation messages. I don't have a fix, or even know what the problem is yet, but trying to keep it working.

[–] jerry@fedia.io 6 points 9 months ago

So far, we are the only ones having the problem. It is almost certainly because we have many more accounts/activity than others. The current problem may be a rabbitmq bug

[–] jerry@fedia.io 7 points 9 months ago (1 children)

it was indeed broken again. should be fixed now

[–] jerry@fedia.io 1 points 9 months ago (1 children)

The modlog function in lemmy/mbin can be a bit confusing. I don't think the moderation actions of one instance propagate to the modlog of another instance. I think that is only really problematic in cases where the moderation action was taken on the home instance of a magazine/community.

[–] jerry@fedia.io 2 points 9 months ago (4 children)

thanks. I think the name of the instance is the most important part - I can test of federation with them to see what went wrong.

[–] jerry@fedia.io 1 points 9 months ago (8 children)

At the moment, there isn't anything going on that would cause that. Can you tell me which other instance/which thread so i can try to troubleshoot?

[–] jerry@fedia.io 3 points 9 months ago

I am investigating…

[–] jerry@fedia.io 1 points 9 months ago (1 children)

Did it take a long time before that error came back?

 

First, I want to thank those who pointed me to mbin. I spent about 14 hours today with help from the mbin team on and off and found/fixed many problems.

Second, this post may not be very intelligible. I’ve been awake for a very long time.

In transparency, at one point today we were pretty convinced that fedia.io had been hacked. But it turns out that sometimes companies go out of business, cancel their servers but leave their domain pointing to that server. And sometimes a person like me is the next person to rent a server with that former customers IP address and then hilarity ensues when your nascent software recognizes that domain is pointing to itself and starts remaining links using the former company’s domain.

During the time I thought it was hacked, I rebuilt it. In doing so, it appears that I’ve fixed many issues that accumulated over the months as kbin went through successive updates.

We did also find a nagging bug in the markdown parser Fedia was running, which caused many (perhaps most) of the remaining error 500’s. I am not clear whether that is fixed now, but if not, we know where to look.

Based on my experience today with the mbin team and the progress of fixing issues, I am fully retracting my intention to shut down Fedia.io and will start working with the mbin team on some performance issues.

Thanks for the patience, everyone.

 

While I’m skeptical there are many people still using fedia.io, I have been continue to try to fix the error 500 problems. These errors are caused by kbin thinking it stored database records with pointers to images (like the avatar image, images attached to posts and the like) but not actually finishing. So when certain records are pulled up (magazines or posts or users) where there is a missing image record, an error 500 exists. I’ve continued to find more instances of this and frankly I don’t know what is causing the problem. As a mitigation, I have been running cron jobs every 15 minutes to correct these broken links. This morning, I believe I chased down the cause of the last remaining disconnect and have that being fixed as well.

I suspect there are still other problems, like replies appearing twice. I will try to figure out what is going on with those as I can.

 

After really hosing things up yesterday, I seem to have solved at least some of the image problems. Not all of them. I rebuilt the mercure and messenger configs from the ground up using the most recent bare metal config templates and that seems to have helped. Also, I continue to live on the edge with the development channel of the kbin repo - there were a number of fixes contributed that looked like issues we had, so I applied those.

Throughout the day today, the site probably was up and down and all around - that was me trying to get things right.

Let's see what I can get done tomorrow while I'm off work and the rest of my family is out of the house.

 

The issue with broken images seems to be getting far worse, and that is the cause of the error 500's. kbin doesn't just skip a missing image, it just dies when trying to render a page. If I don't null out the incorrect database entries often enough, darn near everything starts breaking. I have yet to figure out why this is happening. Tomorrow, my family will be back to work and I'll have some alone time and this is what I intend to track down.

Also, I upgraded to the latest development code, so there may yet be new and exciting bugs.

 

I appreciate your continued patience. Today I applied a few additional fixes, including to nginx, which were contributing to the error 500’s. The good news is that the only errors I see in the kbin app logs are related to inbound post federation from lemmy-based instances. I am sure there are many more issues to, but I’m knocking them down one at a time.

 

I am hesitant to say that I have good news or that I have fixed anything... I honestly don't know if there is anyone left still using this site after all the problems we've had.

That said, I noticed that there were some very recent (as in yesterday/today) commits to the development branch of kbin that look like they address the issues that fedia has been experiencing. I moved fedia to the development branch and reloaded some things, and so far, no error 500's.

But, caveat emptor, we are on the development branch of early-in-its-lifecycle open source software.

 

Due to an unplanned sinkectomy today, I didn't have time to spend on fedia as I planned. Tomorrow, I will carve out some time to look through recent code commits to see if any would address the issues we're experiencing.

 

So here's the deal with kbin: kbin uses of symphony messenger processes, which are roughly equivalent to sidekiq in mastodon.

After I moved fedia from the docker hosted environment to a bare metal instance, I had all manner of database issues - the dump and reload didn't work well, creating many duplicate records. That caused the messenger services to die and the queue of activitypub records to process grew huge. Restarting the messenger service worked, however it would never finish, so I increased the number of messenger workers to 16. That kept the queue nice and clean.

HOWEVER, it appears that running multiple messenger processes creates race conditions where things like images ids are created and assigned to different entity records (like posts) but there is no actual image record created, so when kbin goes to draw a page, it runs a complex query to pull magazine info, post info, comments info, user info and all of their respective images. Those records LOOK like they have an image, but there is no actual image, and so kbin says 💩​ I ain't working and gives the wonderful 500 error.

Setting the messenger services back to 1 seems to be at least not be making the problem worse, but now I have to go find all the broken database record linkages.

 

The site is mostly functioning and respectably fast again.

There are two lingering issues: periodically, federation stops and has to be manually restarted. I have yet to find a good way to automate that or fix the cause.

Some magazines return error 500. I believe this to be a referential integrity issue in the database, so trying to sort that out.

I hope to have these issues fixed over the next few days. Thank you for understanding and for your patience.

If you like fedia.io and want to support it, you can donate using one of the links here:

Patreon: https://www.patreon.com/infosecexchange
Ko-Fi: https://ko-fi.com/infosecexchange
in addition to the existing means:
Librepay: https://liberapay.com/Infosec.exchange/
PayPal: jerry@infosec.exchange

2
2 way cat scan (media.fedia.io)
 
 

Fedia has issues. Some of them I can fix, some of them I can't yet fix. The server's CPUs are maxed with caddy federation requests, probably among other issues.

I am going to unroll the instance into a few separate systems: One for nginx and the php/symphony app, on for the database, and (perhaps) one for rabbitmq and redis. I am also hopeful I can separatre nginx and php-fpm onto separate hosts, but it's not clear how hard or easy that will be.

I'll be taking the server offline for a few hours Saturday morning (eastern time) to do the rebuild/scale out. I'll try to minimize the downtime, though.

Many apologies for the unstable and buggy service here.

 

I believe all of the image problems are solved now.

I understand that there is apparently some issues with remote content. Note that kbin.social and some others may have gone into lockdown mode due to load.

Speaking of load, another reason why some federation doesn't appear to work is that the server is pegged. 32 threads of 90% CPU usage.

I bought 4 more servers and I am going to be giving docker swarm a try to spread the load out across 5 systems.

Wish me luck.

(each system is a Hetzner EX101 Intel i9 13k with 64GB of DDR5 ECC memory and a raid 1 nvme v4 SSD - so pretty fast)

view more: ‹ prev next ›