this post was submitted on 08 Apr 2026
97 points (94.5% liked)

Dull Men's Club

3878 readers
212 users here now

An unofficial chapter of the popular Dull Men's Club.

https://dullmensclub.com/

1. Relevant commentary on your own dull life. Posts should be about your own dull, lived experience. This is our most important rule. Direct questions, random thoughts, comment baiting, advice seeking, many uses of "discuss" rarely comply with this rule.

2. Original, Fresh, Meaningful Content.

3. Avoid repetitive topics.

4. This is not a search engine
Use a search engine, a tradesperson, Reddit, friends, a specialist Facebook group, apps, Wikipedia, an AI chat, a reverse image search etc. to answer simple questions or identify objects. Also see rule 1, “comment baiting”.

There are a number of content specific communities with subject matter experts who can help you.

Some other communities to consider before posting:

5. Keep it dull. If it puts us to sleep, it’s on the right track. Examples of likely not dull: jokes, gross stuff (including toes), politics, religion, royalty, illness or injury, killing things for fun, or promotional content. Feel free to post these elsewhere.

6. No hate speech, sexism, or bullying No sexism, hate speech, degrading or excessively foul language, or other harmful language. No othering or dehumanizing of anyone or negativity towards any gender identity.

7. Proofread before posting. Use good grammar and punctuation. Avoid useless phrases. Some examples: - starting a post with "So" - starting a post with pointless phrases, like "I hope this is allowed" or “this is my first post” Only share good quality, cropped images. Do not share screenshots of images; share the original image.

.

founded 1 year ago
MODERATORS
 

"Is this the one?"
"Yeah, that's the one."
Yank!

As the last of the servers died with a pathetic beep, I think I heard the poor electrician, still holding the unlabeled end of the rack's power line, invoke the name of Jesus.

(Obviously dramatized, but mostly accurate.)

The virtualization servers came back online with some fuss, but they at least look functional. My tasks for the day are set: wade through about 400 error messages, verify the functionality and integrity of 117 virtual machines, restore backups as needed, and verify the SMART status on every physical hard drive.

(edit 1) High Availability tried to migrate all of one host's VMs to the other, but it isn't worth much if both HA hosts are on the same circuit and die within seconds of each other. Now all but a few VMs are running on a single host.

(edit 2) Some of the PhDs are angry because their long-running ML projects got interrupted. They didn't set up checkpoints or live backups, so entirely their fault.

(edit 3) Five hours later, only one VM needed manual intervention (apart from migrating the VMs back to their original hosts), and all the hard drives are in good condition for their age. This turned out to be a really boring disaster.

top 11 comments
sorted by: hot top controversial new old

In 1998 I worked for a local ISP and would occasionally work the late night shift by myself since we offered 24/7 phone support. It was a great job... go to work and spend hours surfing the internet at insanely fast speeds. One night a huge thunderstorm hit and the lights went out, my computer shut down and then about 3 seconds later it came back. The whole time I could still hear the server room humming along. I thought to myself "battery backups are amazing" and didn't give it another thought. Until about 20 minutes later when that server room hum went silent. I was dumbstruck. My computer was still on. The lights were still on. But the whole internet for our region came through that room and it was no longer on. I had no idea what to do... I sat there for anywhere between 5 seconds to 10 minutes trying to figure out what my first step should be and then the phone rang. I answered it expecting an upset customer wanting to know why her Netscape wasn't working, but instead it was my boss, the owner of the company. He never called in, and he sounded chipper and said that he just wanted to make sure everything was ok with the storm that had just come through. I stammered "Uh, so, you don't know what's going on?" He lived about 30 minutes away but he was in the office in 10 minutes and had about 150 feet of extension cords. We ran the extension cords over to the outlets that were powered by the building's generator and got them hooked up to the server room to get it back online.

I've never experienced a worse silence in my life since.

[–] redlemace@lemmy.world 13 points 19 hours ago

true silence i experienced once in an anechoic chamber.

Scarriest silence i've ever heard: a full power-down of a server room i had spent many hours in.

[–] 1995ToyotaCorolla@lemmy.world 5 points 18 hours ago

You just triggered my PTSD. We had a leg of our three phase burn out in juuuust the right way where our UPS wouldn't accept the dirty power but it didn't trigger the backup generator. The electrician didn't arrive in time to manually engage the transfer switch, and the batteries ran down on the ups before we could get everything safely shut down. I don't think I've encountered a quiet quite like that since

[–] FauxLiving@lemmy.world 3 points 18 hours ago

"You hear that? That's the sound of broken SLAs."

[–] Monument@piefed.world 4 points 19 hours ago

Boring disasters are the best disasters.

[–] ramble81@lemmy.zip 8 points 23 hours ago

I’ve migrated two Datacenters and the sound when the last server goes offline is something else. You know it’s supposed to be quiet, but your mind still races for a bit, thinking there’s something wrong.

[–] SpikesOtherDog@ani.social 3 points 20 hours ago (1 children)

Yikes! Was the entire rack plugged into a single UPS?

[–] rtxn@lemmy.world 7 points 19 hours ago

The entire server room is on a single UPS. The rack itself had to be powered down and disconnected to extend its PDUs with C19 sockets.

It's not a very good setup, but (like always) properly fixing it is not worth the cost and the downtime.

Unexpected shutdowns of an entire HA cluster can be fun.

But it “shouldn’t” cause any problems, so mark it down as a test.

[–] Im_old@lemmy.world 8 points 1 day ago

Yeah, that bowel-loosening sensation as your mind races through all the things that could be broken now

Haha! High availability my ass! Made me chuckle 🤭