Its all dark in the NOC......

'twas the hour before free calls and all through the NOC, not a disk drive was whirring, less life than a rock.

Yep, its the dreaded power cut at TCP Towers.... The first we knew about it was all of the lights going out, the UPS beeping at us and the non-essential machines switching off thanks to being on a unprotected supply. Since some of those machines run Win95, this was almost a blessing in disguise.

Half an hour later we took the servers down gracefully; just before the UPS decided to sulk when the power had run away for too long. Until that point, none of our customers had noticed the problem.....

They noticed after the UPS started to sulk. Some of the comments John and I made were, erm, interesting. Some of the comments from customers were even better. The rest of this are the comments we and customers made. We also ran a sweep stakes for when the customers *really* started to call us to report a fault (which I won :)

Anyway, to the comments.

	"I like it dark, but not this fucking dark"
	John, after he realised what would happen when 7pm (the start of the free local calls here) came around.
	"Bugger, down to cold coffee"
	Me, on finishing off my last can of mountain dew
	"One problem of the NOC being dark; wandering into all of the booby traps left for customers"
	Me
	"<whine>I can't point and drool on the internet...</whine>"
	Me, giving an example of how some users would react to the outage.

Some of the customer comments were a little more interesting:

	Luser: What does it mean when all of the lines are ringing out ? Tech: It means that we have a power cut sir
	Tech: ...and so the power will be out for a least a couple more hours Luser: Okay, I'll just crack open a bottle of wine then Tech: Sounds like a good idea to me
	Tech: ....thank to this, you will not be able to access our mail or news servers. Luser: Its the first time I've tried EMail and I thought that I was doing something wrong. Tech: <thinks>Perhaps someone is trying to tell you something</thinks> It is out of our control sir. If you can try again later.....
	Luser: Since I can't log in, could you check my EMail for me ? Tech: No sir, I'm afraid I can't as all of our servers are down.
	Luser: What do you mean you can't help me with Windows 95 support now ? Tech: Sir, as I said, we are in the middle of a power cut and I can't see the phone, let alone documentation on Windows95. Please call back tomorrow.

The drive with our news spool on also decided that it wasn't quite sure about one or two blocks on its drive.... a couple of times every second. Okay, fair enough. We just shut the news server down, mounted the drive read only, brought the news server up and throttled it with a nice message telling customers to read our announcements group for the fault reports. This can almost be considered a good state of affairs. Customers can read Usenet, but they can not post to it :) If only AOL could do the same. The only real problem is the backlog of news we're going to get and the fact that TCP staff can't read new news or post :(

Posted on 15th Feb 1996