“Is this the one?”
“Yeah, that’s the one.”
Yank!
As the last of the servers died with a pathetic beep, I think I heard the poor electrician, still holding the unlabeled end of the rack’s power line, invoke the name of Jesus.
(Obviously dramatized, but mostly accurate.)
The virtualization servers came back online with some fuss, but they at least look functional. My tasks for the day are set: wade through about 400 error messages, verify the functionality and integrity of 117 virtual machines, restore backups as needed, and verify the SMART status on every physical hard drive.
(edit 1) High Availability tried to migrate all of one host’s VMs to the other, but it isn’t worth much if both HA hosts are on the same circuit and die within seconds of each other. Now all but a few VMs are running on a single host.
(edit 2) Some of the PhDs are angry because their long-running ML projects got interrupted. They didn’t set up checkpoints or live backups, so entirely their fault.
(edit 3) Five hours later, only one VM needed manual intervention (apart from migrating the VMs back to their original hosts), and all the hard drives are in good condition for their age. This turned out to be a really boring disaster.
true silence i experienced once in an anechoic chamber.
Scarriest silence i’ve ever heard: a full power-down of a server room i had spent many hours in.
The most concerning sensory deprivation I’ve ever encountered was when Katrina happened. I wasn’t in any of the disaster areas, but it still hit hard where I lived.
When I woke up, I literally couldn’t see anything. The power had gone out and I guess the weather was still cloudy enough that no sunlight penetrated my blinds. Also, it was intensely quiet; I was still young at the time so I didn’t have real servers, just always-on towers, but my stuff was on UPS’ so I was accustomed to at least hearing their beeps if it was out long enough for them to notice.
In this situation, I suppose it had been out long enough to drain their batteries and I had slept deeply enough that the beeps didn’t disturb me.
Also, the house had central HVAC and I was used to hearing it run much of the time. Between that, the lack of system fans, the lack of beeping, and not being able to tell whether my eyes were open, it was a disorienting way to wake up.