The Server Crisis of May/June 2008

[admin]

Welcome back ladies and gentleman.

The server, as you may have read, has been down for more than 30 hours due to a fire at the datacenter in which this server was hosted. More than 9000 servers were affected, with many more sites down including b3ta.com. You can read the whole story here, including the latest updates:

http://service-update.theplanet.com/

Through a bit of luck, our server was one of the first to be brought back online. Many servers are still down and some 3000 will still be down in 12 hours time.

Anyway, now that the server is back, obviously things will need to be done to ensure that a downtime of this magnitude does not happen again, and I've already started on new contingency plans. The extremely bad luck about this incident was that several of the backup plans I put into action, through various different reasons, all failed at the same time. Otherwise, the site would have been back online much sooner.

Anyway, sorry for all the down time and hopefully thing will start to get back to normal soon.

[BR7]

Nice to see the forum back in action

[blutach]

Thanks admin and for providing us with a link to the info while it was down.

Regards

[soup]

Thank you for the info, I am glad everything is back up. My thanks to everyone involved in making that happen.

[atifsh]

i missed this place...

[admin]

It looks like no data was lost, which is good. The downtime felt like a week for me, even though it was only 35 hours or so.

I'm still having trouble occasionally getting to some sites since my ISP is caching the DNS entries for the domain name (so it's still pointing to the "emergency" server). Hopefully things will clear up in a few hours time.

We're actually lucky that the site is back so "soon" as the server was located in the second floor "phase 2" of the racks. The highest profile site affected, b3ta.com, is actually still down and it looks like it could be another 10 hours before it and other "phase 1" servers can come back.

[Wombler]

That was a pretty major outage though.

Very unusual for so much to go down all at once and for so long but it's easy to understand how when you read the circumstances.


Wombler

[PurpleDemon]

Thank goodness it's back up. I missed everyone. At first I thought this forum disappeared like other things. I am relieved it's back.

Funny how crazy things like this happen on the 1st day of the month and what a crazy weekend for major fires.

Thank You to everyone that kept us in the know on the steps to be back. I never knew how big the operation with the servers were till now.

[cynthia]

Took even more time for the DNS to find the adress here in Sweden.

[doctorhardware]

With the short in the wiring in the conduit. The server data center, is extremely lucky that the only thing that was lost was the power coming into the building. There had to be a massive power surge when things went wrong. I saw a ComEd substation blow up. An employee was drilling core samples and hit 500,000 volt line. The truck had to lift up about 5 feet into the air. It was dam loud.

[admin]

There was actually a large explosion within the datacenter that knocked out 3 walls and destroyed the electrical equipments, at least on the first floor. The networking and servers weren't damaged on either floor, but the electrical wiring on the first floor was damaged. Luckily, this server was on the second floor. They are using a temp solution of portable generators to power the first floor until they can rebuild the electrical room in the next week or two. There are some photos of the outside of the datacenter, with all the portable generators in view:

http://www.prisontalk.com/forums/showthread.php?t=340505

Basically this outage was always going to be a several day affair, and the fact that most of the servers were on only one and a half days into the outage means that the web host (The Planet) does deserve some praise. However, why a blown transformer on the outside caused so much damage on the inside is another issue ...

Anyway, I've already ordered another equally powerful server at another datacenter, so the idea is that if one server goes down again, I will switch the sites to the other server in hopefully around 1 hour. I do have another server already in another datacenter that wasn't affected, unfortunately the server wasn't powerful enough to host the site - this is the server that I will replace with the more powerful one that's currently being built. In addition, there is an off-site automated backup system in place (it has been for some while now), so data loss is not as big a problem as before (both servers will also now employ RAID mirror arrays, to mitigate against single hard-disk damage).

[doctorhardware]

Road trip. Admin do you think that they would mind if I borrowed one of the generators.

[uufta]

Thought for sure it went up with Universal Studios, and we were doomed to be homeless..nice to have our home back

[jamoskeag3]

Glad to see that THE FORUM IS BACK!!