this post was submitted on 19 Jul 2024
71 points (94.9% liked)
Sysadmin
7634 readers
19 users here now
A community dedicated to the profession of IT Systems Administration
No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I was with a large organization as tech support and upper IT pushed out an update that corrupted everybody's certificates to log into the network. Imagine having to talk 40k users, who most of them whined and bitched to us about having to do all this work to fix the computer, through a removal of the old certificate, reboot, get the new one after logging in with a backup account, rebooting again and verifying that they can log in. Each computer was about 20-40 minutes to get done. We only had about 50 of us working at peak hours. It took about 2 months of non stop calls to get them fixed.
YIKES ... I got one worse still ... I was NOC at a company where one of my friends from Desktop services made a mistake pushing hard drive encryption and basically corrupted the hard drive of a large number em laptops. It wasn't everyone thank god because they were rolling it out in stages ... but it was THOUSANDS and there was no real way to get it back. Every single one had to get re-imaged.