this post was submitted on 22 Jul 2024
579 points (97.5% liked)
Technology
58083 readers
3123 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.
And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.
None of these things are used in actual server operations.
And it's not much more difficult to fix on Windows, except for the scale of the problem.
Good luck doing that remotely. Which is the sole problem with this most recent CrowdStrike bug.
Anybody who doesn't already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.
And on Windows you booted in safe mode and removed one file. What's the point of your post?
If I ran a computer lab that wasn't already net booted, I'd use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.
There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.