Welcome to August's Server update!
Hope everyone had a lovely July. I'm writing this the next day after back to back weddings, so please bare with me as my head is a bit tender and my bank balance a fair bit lighter.
Straight into it, I've been working on giving Zippy a web based GUI for a while. I've finally settled on a system that works, with a login system based on an OTP system, as you can see in this screenshot:
Once a moderator has logged in, they will be able to see their communities, as well as interact with the functionality of Zippy:
As of right now only RSS feeds and Welcome messages work fully (add, remove, edit), but when I do get some more time I'll be fixing the rest of it up so that all the commands mods can currently do in Zippy can be done by the web ui. Once it is ready and working, I'll release the link to the page, while also updating all the of the messaging we use so that new mods will know this exists.
In technical news, @gazby@lemmy.zip noticed that we were coming under a sneak attack originating in Vietnam, where someone was attempting to aggressively scrape us again. Thankfully, with all the work Gazby has done in getting the monitoring systems we use in place and up to scratch, we negated it fairly quickly, and was able to put a Cloudflare managed challenge in place to prevent the bad actors. This peaked at 800,000 attempts in 24 hours, with only 20 or so real solves. I only managed to catch this screenshot at a later point but it gives the idea of how much traffic gets thrown at us sometimes:
This is the ASN we've added to our challenge for those interested.
Unlike previously when it was just the main site being scraped, this was over all our subdomains so various front ends were falling over from the amount of traffic they were seeing. Thankfully with the challenge in place this has stopped and we've seen immediate and marked improvement all around.
In other technical challenges, we were having big issues with Backblaze and them not accepting our backups seemingly randomly. Images have continued to work fine, but we narrowed it down to issues with large file uploads. We've swapped to Cloudflare's storage solution temporarily to try it out, and so far no issues. Given we had to use a backup not so long ago, we absolutely want to get that process right.
In terms of other server performance improvements - with the settling of the lemm.ee migration, we've seen way less server lockups from settings imports - phew! We are always on the lookout for a beefier server, but the limit is funding. We'd love to have a faster server that can speed up the site, but we also want to make sure we're spending donations wisely and not just on shiny shiny. Speaking of donations..
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
Time for pictures!
CPU over the last 30 days:
Much nice than last month's usage!
RAM:
RAM usage over last 30 days:
Disk space:
Nginx connections:
5xx errors:
(Spikes line up with the scrape activity mostly)
Postgres DB size:
Images:
Yes, that is now a Tb of images.
Cloudflare overview:
Requests:
Bandwidth:
Unique visitors:
Regional traffic:
Here is a bonus pic just to show how much Vietnam was scraping us:
On that note, I'm off to attempt to catch up on some sleep.
Finally, there's some big news coming soon (in a good way hopefully!) - for now, I would humbly ask that if you are enjoying any of our .zip sites, please consider a small donation at either of the links above. Any amount really goes a long way and helps keep the sites up. We don't, and never will, personally profit from this project. That's not why we do this.
We do this because we believe in a free and open internet free from dangerous algorithms and constant advertising, no matter what governments around the world are currently trying to do to destroy that.
Hope you all have a peaceful August, and I'll be back on the 15th over at !home@piefed.zip for our Piefed.zip server update :)
Thanks,
Demigodrick
I don't use it currently (except on one of the front ends we use as a test) but I believe the easiest way would be to use nginx to separate the paths you want to pass traffic to, so it doesnt apply to api traffic etc, only the ui. I use traefik for piefed so I don't know how it integrates in there yet, but I know a few people have got it working for lemmy by using the nginx paths.
There is also go-away as an alternative option.
I have made extensive use of cloudflares challenges too
There's only one solution - you'll have to move countries 😉
Totally missed that it was happening this year 🙈
Because I too am a billionaire fascist.
Or possibly slightly more likely its just pulling rss feeds.
Great, glad it worked 😁
Thanks Blaze! Will run that asap!
We're at about 1000 now, looks like a lot we're waiting for .ee to shutdown. I guess quite a few went to piefed too
Hello all!
What a weird month it has been.
The biggest piece of news is the impending closure of lemm.ee. We've been fortunate enough to be recommended by a lot of people so have taken in a large amount lemmeegrants/refulemm.ees.
If you have come from lemm.ee and not seen our welcome post yet, then please Click Here!
Since the closure announcement of Lemm.ee we've taken in over 800 new accounts. We have also been made the default instance of the Voyager app, which has seen some new users directed our way.
This has been great for helping to grow the instance, but has really introduced some performance flaws when a user imports their account settings. For most people this is usually not noticeable, but if you import a very large settings file, it can actually bring the server to a standstill. For example, here's one that brought the server to a crawl for 30 minutes:
Therefore please really consider if you do need to import your file.
Thankfully the other 99% of the time the server is performing great even with an extra 800 monthly users.
You may remember from last month we were getting a bit crushed under what we assumed was scraper traffic. After investigations, tweaks, and implementing cloudflare challenges on suspected badly behaving ASNs, we've seen a massive reduction in this, and almost an elimination of 5xx errors (except when, you know, I break something!)
So outside of the imports, server performance is looking much better. We will see how things get on with any extra traffic we do get, and we will upgrade if we need to.
We also created a Piefed server - Piefed.zip! - More info can be found in this thread
Finally, we also updated to Lemmy 0.19.12. This went without issue, but took a little longer than expected sorry!
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
This the actual fun stuff.
CPU over last 30 days (and yes, the spikes are imports!)
RAM usage over last 30 days (also affected by imports)
Disk space (really not that exciting)
Nginx connections
5xx errors
Lemmy database size
Images:
Cloudflare overview:
Requests:
Bandwidth:
Unique visitors:
Traffic:
Sorry this one is a bit briefer than usual - off on a little holiday so lots of packing to do, and not much time!
Hope everyone has a nice July!
Thanks
Demigodrick
Hello all!
Firstly, apologies I missed last month's update. As I will go in to during this month's update, the train has not been smooth sailing, the ship missed the station - for a little while, trying to uncover what has gone wrong has been like unraveling a can of worms that has gone off the tracks.
Because of the length of this update, I'm leaving off new communities for this month, but feel free to pop any in the comments you'd like to share!
So, I'll begin at the 0.19.11 update for Lemmy. I'd planned in some time for the update, and to begin with things went fairly normally, with usual server updates etc. Then it came time to update Lemmy itself to 0.19.11 - however, instead of the ~10 minutes of suggested downtime, it quickly became apparent that something wasn't quite right. The logs gave no indication of anything happening, and after an hour it became apparent that something hadn't worked. With not much to go on, and all the various logs showing that absolutely nothing had apparently happened, I cancelled the upgrade process and tweaked the logs so I'd get a bit more information than the standard logs usually show. This turned out to be an internal DNS issue, where Lemmy wasn't talking to the database.
So, onwards again I fixed that issue and reapplied the update, to which it had appeared to start working. Excitedly, I give it ten minutes as advised, to come back and see that it was still going.. and going.. and going.. and two hours later it was saying something was still happening. Well, by this point I'd run out of time and trusted in the process that it would resolve itself. I had unfortunately had to go to sleep as I had work the next morning, and a baby to look after.
By the next morning, still nothing. However, I can't SSH in to the server on my phone (nor would I ever want to in case it was stolen etc) and all the backend stuff is behind layers of protection, so the site was down for the day while I had to do real world work nonsense.
By the time I'd got home, it still thought it was ongoing, so I made the decision that likely something was broken and cancelled the update. Of course, letting something go to town for hours on the database meant it was probably ruined, or would require completely investigating - ain't nobody got time for that. So I crossed my fingers and hoped my backup solution had worked which it did with flying colours. I restored the database and the site came back to life.
Of course, I don't like running old versions, so I rescheduled which went much much smoother the second time around, taking 8 minutes. Whew.
For a few days, everything looked great. Metrics looked spot on and the site was running like it always was, and then bam - the server completely locked up:
A restart did eventually fix this, and things returned to normal, but not quite the same.
On the 24th April, I started getting reports of the site returning various 502 and 504 errors, and also issues with images loading. Broken images has been a feature of the image proxy for a while, but this was on another scale, with sometimes whole pages of broken images and a connection timeout on every refresh. I was, in hindsight, very out of my depth here.
I started trying to see what the cause of the timeouts where, and it became apparent quickly that the site was being hit by previously-unseen levels of traffic:
On the above graph, the left hand side was the usual amount of traffic we'd get, and on the right was the new levels we were receiving. This was likely causing the server to effectively be DDOS'd, and uncovering issues with the performance of the server. For a comparison of the usual amount of traffic we get, here's a graph from the last server update:
Many configs were tweaked. Much database performance monitoring took place. Things were turned off and back on indiscriminately. Unfortunately not much worked.
It was at this point our new Admin, Gazby, joined the team - and quickly got to work diagnosing and fixing so many of the issues.
Some of the changes include much better backups, reduced latency in image storage (the images moved from the USA to EU storage), images being served from i.lemmy.zip to allow for better caching to reduce the load on the server, complete review and tweak of all the configs and server setup, and a detailed list of things we need to work on going forwards (plus no doubt lots of things I am forgetting).
This has really helped to stabilize the site, and while we can see it's not 100% perfect, the amount of 502/504 errors should have massively reduced, and the site should be almost as it used to be.
One of things Gazby has also worked on is reporting and insights into the server, and so for example here's a graph showing the 504 errors today:
We've got some plans on the horizon we're working on which should increase the performance of the site, which possibly/probably will include a server move to rule out hardware issues or latent config issues we can't find.
Whatever triggered these issues I am still not sure - it looks like it could be scrapers/AI bots hammering the site, and so we've put some measures in place here. If you use the old front end then you may have noticed a cloudflare challenge to try and prevent the server being overloaded - also a prewarning, the old front end is no longer maintained, and unless someone steps in, it is not compatible with the next version of Lemmy. Therefore at that point it will be retired unless someone works on it to bring it back in line. It is also the cause of a LOT of the server traffic problems, so it probably isn't too much of a bad thing. Maybe someone will rewrite it to be better :)
As it stands, things still aren't perfect - we had an issue where the bit of the server that actually directs you to the right place got overwhelmed, and so we put a fix in for that, but they are hopefully a lot better than they were a couple of months ago.
Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!
We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.
If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.
We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:
We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!
Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.
I know you've come for the shiny pictures, so here you go!
CPU over last 30 days:
RAM over last 30 days:
Disk space used:
Here's a few new ones for you! Lemmy DB size:
Images database size:
Here's our current actual images stored:
Here's the cloudflare overview:
Here's requests:
Bandwidth:
And here is visitors:
And finally, traffic by country (mostly federation traffic remember!)
So hopefully that fills everyone in on where we're up to, and what we're working on, but if you have any questions please ask away below!
One final thing - on the 10th June, Lemmy.zip turns two years old!! 🥳 🎉 I'm hoping to do something nice for it, similar to last years (which is here if you haven't seen it!) - but a quick thank you to everyone who has been part of the ride so far!
Thanks
Demigodrick
"Oi, 'scuse me mate, have you got a wank loicence for that file system?"