crashdoom

EDIT Aug 2, 1:53 PM US Mountain Time: Maintenance has now concluded and all services are back online!

We're preparing the server rack for an additional Proxmox node to be added to support HA failover and have identified the need to offline and move some equipment around to complete this work.

As such, we'll have a temporary downtime for up to 1 hour from 1 PM US Mountain Time, today August 2nd, or 1 hour from this post. Though, we expect the work to update the node, move it, and bring it back online to take no longer than 30 minutes.

We're hoping these changes will help us maintain a more stable service going forward, spread out the Sidekiq processes (the ones that receive / send data across the Fediverse), and allow us to perform maintenance as necessary without having to offline the service in the future.

We apologize for the short notice as we had hoped to perform this maintenance without any downtime.

Maintenance window: 1 PM - 2 PM US Mountain Time, August 2nd, 2023

ETA: 30 minutes to 1 hour

During the maintenance window, the following services will be unavailable:

• pawb.fun
• pawb.social
• furry.engineer

We’ll have updates throughout the maintenance window available at https://t.me/pawbsocial.

• Instance: burggit.moe
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: CSAM / Loli / Rule 1

• Instance: federated.fun
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Trolling, transphobia, hate speech, unmoderated / single-user instance

• Instance: ligma.pro
• Type: Defederation
• Affects: Pawb.Social, furry.engineer, pawb.fun
• Reason: Trolling, harassment, gaslighting admin

Cleared The Shifting Oubliettes of Lyhe Ghiah with a group of four, hitting every single exclamation mark (though two were secret sweeper trolls)!

Love the fact the devs are willing to troll you. FFXIV is awesome XD

Around 11:30 PM, one of the core GCFI breakers tripped resulting in the UPS array running until around just after midnight before dropping entirely.

We had not set up monitoring on the APC UPS as we hadn’t anticipated that as a failure point due to the reliability of the Colorado electrical grid, and didn’t anticipate the breaker itself flipping after months of continued use.

All services should be online again and working to catch up on any dropped content overnight, and we apologize again for the inconvenience.

We’ll be trying to identify the root cause throughout the day, working on the electrical connections, and setting up alerting through the APC UPS to detect and persistent A/C loss.

EDIT Maintenance Complete (7/6 8:41 AM): We’ve finished troubleshooting with the ISP modem (and stumbled onto fixing the issue somehow), so we’ve brought pawb.social back online.

The Mastodon upgrade has completed and we’re now running Mastodon+Glitch 4.1.3!

…What’s Glitch? Glitch-soc is a fork of Mastodon that enables a ton of cool new features, accessibility options, and more!

You’ll see new things like:

• An option under the three-dots when posting on the web UI to: Enable threading, or keep your post entirely local to your instance (We’ll be using this going forward with an announcement bot on both pawb.fun and furry.engineer to avoid spamming the timeline)

• Formatted toots! (You can now use markdown language in your toots!)

• More skins and flavors of the web-UI, including a high contrast version

Let us know if you have any issues!

Following our previous post about ISP issues, we'll be performing tests on our ISP equipment tomorrow morning (July 6th) from 7 AM US Mountain Time for several hours.

Additionally, we've been made aware of a critical security update for Mastodon that will be released around the same time which we will be applying during the maintenance window.

We're unsure the contents of the security update at this time, but have been told there is no reason to believe it has been exploited in the wild.

During the maintenance window, the following services will be unavailable:

• pawb.fun
• pawb.social
• furry.engineer

We'll have updates throughout the maintenance window available at https://t.me/pawbsocial.

We hope to complete the maintenance as soon as possible and apologize for any inconvenience this causes.

EDIT Mitigated (7/3 10:14 AM): We’ve been temporarily downgraded to 100Mbps to lower the bandwidth stress on the network due to a damaged fiber trunk cable.

The service should be stable now but some requests may take longer than expected due to occassional ping spikes.

An engineer has been scheduled for Wednesday 7/5 and we’ll have an update then. We apologize for any inconvenience.

EDIT (7/3 @ 1:29 AM): The service appears to be mostly functional again and is processing through missed messages, so you may see delays in receiving notifications and feed posts from during the downtime.

We’re still seeing above normal pings and the occasional dropped packets, but the connection seems mostly stable.

We’ll check in again around 10 AM US Mountain Time.

EDIT (7/3 @ 12:52 AM): We've exhausted our troubleshooting at this time and are waiting a repair technician. While the service does appear to be more stable right now, we're still seeing higher than normal pings.

We've created a backup Telegram channel for announcements in the event of the site going offline again: https://t.me/pawbsocial.

As of July 2nd, 2023 at 10:07 PM, we're currently experiencing an intermittent Internet outage. We've reported the issue to our ISP and are awaiting a resolution.

The service may be partially or fully unavailable for short periods during this time.

We apologize for any inconvenience.

Applications are now closed! Thank you to everyone who applied, we’ll be reviewing the applications this week and be in touch soon!

EDIT: Due to issues with sending DMs between Lemmy and Mastodon, if you're on Pawb.Social submitting an application, please send it to @admin@pawb.social via https://pawb.social/create_private_message/126149 instead.

Pawb.Social services have been steadily growing more and more over the past few months, from the debacle of Twitter and the meltdown of Reddit, we've seen users flocking to not only our instances but those of our wider fediverse community.

Seeking Content Moderators

We're seeking to bring on a small group of volunteer Content Moderators to help with addressing reports, identifying content that breaks the Code of Conduct, and assisting users with concerns and issues.

Please consider the following questions:

• Are you friendly, polite, and willing to be approached by users?
• Are you willing to help users on our instances?
• Do you have free time to commit and are you able to stick with it long term (6+ months)?
• Do you work well with others in a team?
• Are you willing to join a Telegram group to coordinate decisions, issue reporting, and communications with the team?
• Have you ever had any moderation experience?
• Which instance would you prefer to moderate? (furry.engineer, pawb.fun, pawb.social, all?)
• Do you understand that reports you handle may result in being exposed to indecent, graphic, or otherwise rule-breaking imagery that may be disturbing?

You may (but are not required to) add additional information to your application giving us a little background on yourself, why you're interested in joining the team, your favorite topic (Lemmy) / hashtag (Mastodon), or anything else you'd like. Not including this will not affect your application, it's purely optional.

How to apply

If the questions haven't scared you away and you're still willing to assist us, you must have been a user on Pawb.Social services for at least one (1) week, have had no previous CoC violations, and have no history of recent (past few years) disruptive or abusive behavior within online communities.

If you'd like to apply from Mastodon, please send a DM to @staffapps@furry.engineer.

If you're on Lemmy and you'd like to apply, click this link to send a DM from your Pawb.Social account: https://pawb.social/create_private_message/126149.

Timeline

Applications will remain open for two (2) weeks and will close on July 14th, at noon US Mountain Time.

We expect to review the applications the following week and reach out to all applicants with the result of their application shortly thereafter.

Thank you for being the awesome part of Pawb.Social!

We've just updated to the latest version of Lemmy which comes with a ton of bug fixes and an updated UI without websockets (yay, easier debugging).

You can find the full changes over at https://github.com/LemmyNet/lemmy/releases/tag/0.18.0.

This also appears to have fixed an issue with connecting to kbin instances. You should now be able to search those up in the same way as other Lemmy communities: !community@domain, but let us know if there's any issues!

Over the past 48 hours, we've received a lot of discourse regarding the upcoming "Project 92" platform created by Facebook and the anti-meta fedi pact.

While we're still in early days of the request for comments, a clear pattern has emerged that users of Pawb.Social services unequivocally do not want their data to be abused by corporate interests, such as those of Facebook.

Over and above the corporate interests, Facebook has also demonstrated a repeated failure to enact meaningful moderation changes to avoid abusive content on their platform, especially misinformation.^[^^1^^][^^2^^][^^3^^]^

"I do wonder if this discussion is a little academic, because I feel like Meta instances are very likely to get defederated for some reason under the current rules, like poor moderation, spam, or generic abuse." ^[^^4^^]^

As pointed out above, excluding the corporate interests, the discussion is somewhat moot due to the concerns regarding moderation which would lead to a Facebook operated instance being inevitably suspended.

For these reasons, Pawb.Social (including furry.engineer and pawb.fun) will suspend any and all Facebook owned, operated, or affiliated instances in the interests of protecting our users and communities.