A few weeks back I received a letter from our local electric utility informing me of scheduled overnight outages due to service work in my neighborhood.  The day came, and past, without the announced outage.  This is not unusual, it has happened before so I didn’t think much of it.

A few days later I received an early morning call that our office email and Internet access were down.  I had not received any NAGIOS alerts overnight but I thought that the utility had finally gotten around to doing the scheduled work and that the power had failed overnight and exhausted the UPSes.

I walked the caller through the UPS restart process and found that they were already running fine.  However, all the equipment in our mail server rack was powered off.  This includes our firewall (which explains the Internet outage ) as well as the NAGIOS monitoring machine (which explains why I did not receive any alerts) and the mail server.  Hmm…  the plot thickens.

Continue reading Never say “It could be worse…” because sometimes it is. →