NYC Datacenter and What it means to customers

[Elochai]

Well let me start off with saying that I know this is not the fault of NFO but I do have questions.

So the Datacenter which should be state of the art and built to take a beating if the world stop turning doesn't really have any way of protecting itself from floods ?. So because of this our servers could lose power as the result of flood water taking out the fuel pumps for the backup generators. Floods are one of the things datacenters are supose to be able to take on. Even where I work they have sub-pumps in the lower levels that will kick in it flooding happens.

Now again I know NFO can't help that sandy coming or that internap Datacenter was not built to take on these types of issues. But if the servers go down, will NFO be providing credit for the down time ?, I know the people renting from me are going to want credit as it is down time and the Datacenter should of been able to take on floods.

For the record I'm not mad or anything if it sounds like it, I am rushing to get to work tho :p. I am just looking for some information and an answer to that question.

Other then that, I'm happy sandy hasn't hit my area ...... Yet

Elochai

[Vanderburg]

In NYC, a flood on Broad St bad enough to take out the diesel pumps is not something that anyone anticipated, I'm sure. The record surge at NYC was 11 feet (In like 1826), but yesterday, it reached 14 feet, so this would be something I'm sure no one would expect.

Long story short, if you would like an SLA credit, please wait until everything is back up and running before making the request and be sure to remember that SLA credits must be at least $1.00, so if you have a $10/month server, we'd have to be out for almost 4 days before you reach $1, so if you can manage to do some math (I know it can be complicated if you have multiple services, etc) then you can know ahead of time if you'll qualify.

[Elochai]

It a VDS and meets the amount and my server not offline yet.

I know situation, all I'm staying is what happened ? Datacenters are built to take on the unseen. Why didn't this one have common sub-pumps in the gen rooms to pump out any incoming water ?

But thanks for letting me know that in the event of down time I can requst a credit as it will be needed if I'm down for a day or more. Like I said, I also got people renting from me as well.

Thanks for the fast reply.

Elochai

[Vanderburg]

Not a problem. I'm really not qualified to answer the "Why?" question and was more or less offering an opinion, not facts. Looking around at various forums, I've seen at least 4 or 5 other data centers from other providers that are out as well. I guess that's why they call it a disaster.

[Elochai]

Wow, I guess the real question to it, is how close was the DC to the surge. Did it get the full effect or just the flooding ?

[dyno]

Here is what seems to be the full message internap sent out:

http://pastebin.com/9Fr2eW6U

One key part, in my opinion, missing from the update provided by NFO:

"The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level." [emphasis mine].

When I read that update it sure seemed a lot more dire than the update provided by NFO. Hardware is failing and it seemed the only hope for services to be restored is for main power to come back since the pumps are destroyed.

We just had one gameserver hosted in NY and were planning on keeping it there. But once we saw that physical damage to the pumps had taken place we took advantage of the NFO's excellent control panel and had a (mostly) smooth transfer to another DC. All config files on our BF3 server were lost but thankfully the transfer was quick and the GUID remained the same.

[Python]

It looks like the NYC data center is officially offline as i just lost contact with my NY VDS,

-Py

[RainMotorsports]

Yeah our VPS is down that runs our Procon, Cod 4 and TS.

I had backed up the VPS to my personal VPS in Chicago and got Procon and TS running now that I know NYC might be out for a bit.

[Elochai]

Yup it down for me and I'm not lucky enough to have the money to have a 2nd VDS in another close area

[anakaoka]

while I agree that InterNap should have put fuel pumps higher , maybe even a smaller tank higher up... s*#t happens right?

What are you hosting that someone in NYC will care about? I'm thinking people close who wanted low latency are pretty much bunkering down...

Just buy a 2nd NFO server (like we did) and move your stuff... it forced us to go to Dallas.

[Elochai]

while I agree that InterNap should have put fuel pumps higher , maybe even a smaller tank higher up... s*#t happens right?

What are you hosting that someone in NYC will care about? I'm thinking people close who wanted low latency are pretty much bunkering down...

Just buy a 2nd NFO server (like we did) and move your stuff... it forced us to go to Dallas.

I'm sorry but I can't argee with you.

1. The main point of a Datacenter is to stay online when the sh!t hits

2. I already told you I don't got the money (unless I don't want my family to have Christmas) for a 2nd VDS. And why the hell would I pay for a 2nd VDS ?, if NFO wants to give me a 2nd VDS till my NY one back then fine, but I'm not going to pay double for a product.

3. NY is the closes Datacenter NFO has for me, I am on the island of Newfoundland. So unless NFO gets Montreal Canada then NY is what I need.

[RainMotorsports]

I know situation, all I'm staying is what happened ? Datacenters are built to take on the unseen. Why didn't this one have common sub-pumps in the gen rooms to pump out any incoming water ?
Elochai

Here is the Data Center - https://maps.google.com/maps?f=q&source ... 80&iwloc=A

Its a building that was there long before Internap was there and if your going to build anything in NYC that is often the primary choice unless you can demolish and develop the entire lot.

Its also in an area hit very very hard and while a purpose built data center probably would have survived on generator power the networking in that area might not have survived a storm like this anyway.

[hiimcody1]

Elochai please open a support request, I will get you a temp VDS for a couple days.

[RainMotorsports]

Also should add to my post about the location of this building.

The fuel pumps would be at tank level. Its is likely between the building code and the pre existing structure these had to be in the basement. The generators are on a higher level and feed off the "reserve" tanks which are in turn kept full by the pumps in the basement with their main supply.

If the tanks had to be in the basement, the pumps belong there with them. Not really a design choice they had. As far as pumping out the basement. You have to have somewhere for the water to go.

[Elochai]

Elochai please open a support request, I will get you a temp VDS for a couple days.

I will do that when I'm off. It won't be much help to me I don't think, but it should keep my client happy and not wanting a refund for the down time.

Also should add to my post about the location of this building.

The fuel pumps would be at tank level. Its is likely between the building code and the pre existing structure these had to be in the basement. The generators are on a higher level and feed off the "reserve" tanks which are in turn kept full by the pumps in the basement with their main supply.

If the tanks had to be in the basement, the pumps belong there with them. Not really a design choice they had. As far as pumping out the basement. You have to have somewhere for the water to go.

It to bad they didn't have the room to redesign parts of that building to better support the generator issues. I believe if they were running the server would be still up. The thing about the network in New York is that it is up and running still. Water won't hurt the fiber and the points for each router or relay's seem to be still running as well so far.

As for pumps I understand that issue with where to pump water, they would of had to do what we do here, our basement is a sub basement (A basement under a basement). We have to pump it up about 2 floors.

Thank you tho as this is the kind of information I was wondering about . I do hope things get better tho for everyone however as the damage to the area is nuts.