Welcome to August's Server update!

Hope everyone had a lovely July. I'm writing this the next day after back to back weddings, so please bare with me as my head is a bit tender and my bank balance a fair bit lighter.

Server Updates

Straight into it, I've been working on giving Zippy a web based GUI for a while. I've finally settled on a system that works, with a login system based on an OTP system, as you can see in this screenshot:

Once a moderator has logged in, they will be able to see their communities, as well as interact with the functionality of Zippy:

As of right now only RSS feeds and Welcome messages work fully (add, remove, edit), but when I do get some more time I'll be fixing the rest of it up so that all the commands mods can currently do in Zippy can be done by the web ui. Once it is ready and working, I'll release the link to the page, while also updating all the of the messaging we use so that new mods will know this exists.

In technical news, @gazby@lemmy.zip noticed that we were coming under a sneak attack originating in Vietnam, where someone was attempting to aggressively scrape us again. Thankfully, with all the work Gazby has done in getting the monitoring systems we use in place and up to scratch, we negated it fairly quickly, and was able to put a Cloudflare managed challenge in place to prevent the bad actors. This peaked at 800,000 attempts in 24 hours, with only 20 or so real solves. I only managed to catch this screenshot at a later point but it gives the idea of how much traffic gets thrown at us sometimes:

This is the ASN we've added to our challenge for those interested.

Unlike previously when it was just the main site being scraped, this was over all our subdomains so various front ends were falling over from the amount of traffic they were seeing. Thankfully with the challenge in place this has stopped and we've seen immediate and marked improvement all around.

In other technical challenges, we were having big issues with Backblaze and them not accepting our backups seemingly randomly. Images have continued to work fine, but we narrowed it down to issues with large file uploads. We've swapped to Cloudflare's storage solution temporarily to try it out, and so far no issues. Given we had to use a backup not so long ago, we absolutely want to get that process right.

In terms of other server performance improvements - with the settling of the lemm.ee migration, we've seen way less server lockups from settings imports - phew! We are always on the lookout for a beefier server, but the limit is funding. We'd love to have a faster server that can speed up the site, but we also want to make sure we're spending donations wisely and not just on shiny shiny. Speaking of donations..

Donations

Lemmy.zip only continues to exist because of the generous donations of its users. The operating cost of Lemmy.zip is over 60 euros a month ($60, £50) and is mostly funded by the community!

We keep all the details around donations on our OpenCollective page, with full transparency around income and expenditure.

If you're enjoying Lemmy.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the site and keeping the virtual lights on.

We've also put up a link to our Ko-Fi page where you can donate via paypal instead of using a card. All Ko-Fi donations will be totalled up and added to OpenCollective each month for transparency. I've added a link in the sidebar, but you can also click the image below to go there:

We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too!

Please remember, traditional social media is only "free" to you because they sell your data. We don't do that - if you want to support independent social websites like this one and you value your privacy, please consider a small donation. It really does help.

Graphs

Time for pictures!

CPU over the last 30 days:

Much nice than last month's usage!

RAM: RAM usage over last 30 days:

Disk space:

Nginx connections:

5xx errors: (Spikes line up with the scrape activity mostly)

Postgres DB size:

Images:

Yes, that is now a Tb of images.

Cloudflare overview:

Requests:

Bandwidth:

Unique visitors:

Regional traffic:

Here is a bonus pic just to show how much Vietnam was scraping us:

On that note, I'm off to attempt to catch up on some sleep.

Finally, there's some big news coming soon (in a good way hopefully!) - for now, I would humbly ask that if you are enjoying any of our .zip sites, please consider a small donation at either of the links above. Any amount really goes a long way and helps keep the sites up. We don't, and never will, personally profit from this project. That's not why we do this.

We do this because we believe in a free and open internet free from dangerous algorithms and constant advertising, no matter what governments around the world are currently trying to do to destroy that.

Hope you all have a peaceful August, and I'll be back on the 15th over at !home@piefed.zip for our Piefed.zip server update :)

Thanks,

Demigodrick