Programming

21924 readers

692 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 2 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

MaungaHikoi@lemmy.nz

UlrikHD@programming.dev

Our first outage from LLM-written code (sketch.dev)

submitted 2 days ago by skip0110@lemmy.zip to c/programming@programming.dev

6 comments fedilink hide all child comments

cross-posted from: https://lemmy.bestiver.se/post/526967

Comments

you are viewing a single comment's thread
view the rest of the comments

[–] deegeese@lemmy.dbzer0.com 49 points 2 days ago* (last edited 2 days ago) (2 children)

I gasped when I saw this:

A bit of discussion indicated that the trigger for the CPU spikes both times was our CEO logging in. We re-deployed to get a clean start, permanently banned him from the service, and moved on.

This is like finding a live grenade under your bed and putting it under the rug.

They found a way to reproduce a system killing bug, and instead of taking the time to understand it, they threw away their test case.

[–] BlazeDaley@lemmy.world 35 points 2 days ago

They contained the impact. Root causing or “understanding” should come after impact mitigation. If needed find a safe way to reproduce the bug without customer impact.

We reverted the refactoring, deployed, un-banned the CEO, and set about analysis.

[–] FizzyOrange@programming.dev 9 points 2 days ago

Yeah me too but if you keep reading they didn't actually "move on" in the way that it sounds.