NYC Datacenter and What it means to customers
-
- This is my homepage
- Posts: 93
- https://www.youtube.com/channel/UC40BgXanDqOYoVCYFDSTfHA
- Joined: Thu Sep 15, 2011 1:23 pm
- Location: Newfoundland
NYC Datacenter and What it means to customers
Well let me start off with saying that I know this is not the fault of NFO but I do have questions.
So the Datacenter which should be state of the art and built to take a beating if the world stop turning doesn't really have any way of protecting itself from floods ?. So because of this our servers could lose power as the result of flood water taking out the fuel pumps for the backup generators. Floods are one of the things datacenters are supose to be able to take on. Even where I work they have sub-pumps in the lower levels that will kick in it flooding happens.
Now again I know NFO can't help that sandy coming or that internap Datacenter was not built to take on these types of issues. But if the servers go down, will NFO be providing credit for the down time ?, I know the people renting from me are going to want credit as it is down time and the Datacenter should of been able to take on floods.
For the record I'm not mad or anything if it sounds like it, I am rushing to get to work tho :p. I am just looking for some information and an answer to that question.
Other then that, I'm happy sandy hasn't hit my area ...... Yet
Elochai
So the Datacenter which should be state of the art and built to take a beating if the world stop turning doesn't really have any way of protecting itself from floods ?. So because of this our servers could lose power as the result of flood water taking out the fuel pumps for the backup generators. Floods are one of the things datacenters are supose to be able to take on. Even where I work they have sub-pumps in the lower levels that will kick in it flooding happens.
Now again I know NFO can't help that sandy coming or that internap Datacenter was not built to take on these types of issues. But if the servers go down, will NFO be providing credit for the down time ?, I know the people renting from me are going to want credit as it is down time and the Datacenter should of been able to take on floods.
For the record I'm not mad or anything if it sounds like it, I am rushing to get to work tho :p. I am just looking for some information and an answer to that question.
Other then that, I'm happy sandy hasn't hit my area ...... Yet
Elochai
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!
- Vanderburg
- Former staff
- Posts: 1253
- Joined: Sat Nov 13, 2010 7:27 am
- Location: Dallas, TX
Re: NYC Datacenter and What it means to customers
In NYC, a flood on Broad St bad enough to take out the diesel pumps is not something that anyone anticipated, I'm sure. The record surge at NYC was 11 feet (In like 1826), but yesterday, it reached 14 feet, so this would be something I'm sure no one would expect.
Long story short, if you would like an SLA credit, please wait until everything is back up and running before making the request and be sure to remember that SLA credits must be at least $1.00, so if you have a $10/month server, we'd have to be out for almost 4 days before you reach $1, so if you can manage to do some math (I know it can be complicated if you have multiple services, etc) then you can know ahead of time if you'll qualify.
Long story short, if you would like an SLA credit, please wait until everything is back up and running before making the request and be sure to remember that SLA credits must be at least $1.00, so if you have a $10/month server, we'd have to be out for almost 4 days before you reach $1, so if you can manage to do some math (I know it can be complicated if you have multiple services, etc) then you can know ahead of time if you'll qualify.
Re: NYC Datacenter and What it means to customers
It a VDS and meets the amount and my server not offline yet.
I know situation, all I'm staying is what happened ? Datacenters are built to take on the unseen. Why didn't this one have common sub-pumps in the gen rooms to pump out any incoming water ?
But thanks for letting me know that in the event of down time I can requst a credit as it will be needed if I'm down for a day or more. Like I said, I also got people renting from me as well.
Thanks for the fast reply.
Elochai
I know situation, all I'm staying is what happened ? Datacenters are built to take on the unseen. Why didn't this one have common sub-pumps in the gen rooms to pump out any incoming water ?
But thanks for letting me know that in the event of down time I can requst a credit as it will be needed if I'm down for a day or more. Like I said, I also got people renting from me as well.
Thanks for the fast reply.
Elochai
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!
- Vanderburg
- Former staff
- Posts: 1253
- Joined: Sat Nov 13, 2010 7:27 am
- Location: Dallas, TX
Re: NYC Datacenter and What it means to customers
Not a problem. I'm really not qualified to answer the "Why?" question and was more or less offering an opinion, not facts. Looking around at various forums, I've seen at least 4 or 5 other data centers from other providers that are out as well. I guess that's why they call it a disaster.
Re: NYC Datacenter and What it means to customers
Wow, I guess the real question to it, is how close was the DC to the surge. Did it get the full effect or just the flooding ?
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!
Re: NYC Datacenter and What it means to customers
Here is what seems to be the full message internap sent out:
http://pastebin.com/9Fr2eW6U
One key part, in my opinion, missing from the update provided by NFO:
"The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level." [emphasis mine].
When I read that update it sure seemed a lot more dire than the update provided by NFO. Hardware is failing and it seemed the only hope for services to be restored is for main power to come back since the pumps are destroyed.
We just had one gameserver hosted in NY and were planning on keeping it there. But once we saw that physical damage to the pumps had taken place we took advantage of the NFO's excellent control panel and had a (mostly) smooth transfer to another DC. All config files on our BF3 server were lost but thankfully the transfer was quick and the GUID remained the same.
http://pastebin.com/9Fr2eW6U
One key part, in my opinion, missing from the update provided by NFO:
"The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level." [emphasis mine].
When I read that update it sure seemed a lot more dire than the update provided by NFO. Hardware is failing and it seemed the only hope for services to be restored is for main power to come back since the pumps are destroyed.
We just had one gameserver hosted in NY and were planning on keeping it there. But once we saw that physical damage to the pumps had taken place we took advantage of the NFO's excellent control panel and had a (mostly) smooth transfer to another DC. All config files on our BF3 server were lost but thankfully the transfer was quick and the GUID remained the same.
Re: NYC Datacenter and What it means to customers
It looks like the NYC data center is officially offline as i just lost contact with my NY VDS,
-Py
-Py
- RainMotorsports
- A semi-regular
- Posts: 21
- Joined: Tue Apr 24, 2012 4:39 am
Re: NYC Datacenter and What it means to customers
Yeah our VPS is down that runs our Procon, Cod 4 and TS.
I had backed up the VPS to my personal VPS in Chicago and got Procon and TS running now that I know NYC might be out for a bit.
I had backed up the VPS to my personal VPS in Chicago and got Procon and TS running now that I know NYC might be out for a bit.
Re: NYC Datacenter and What it means to customers
Yup it down for me and I'm not lucky enough to have the money to have a 2nd VDS in another close area
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!
Re: NYC Datacenter and What it means to customers
while I agree that InterNap should have put fuel pumps higher , maybe even a smaller tank higher up... s*#t happens right?
What are you hosting that someone in NYC will care about? I'm thinking people close who wanted low latency are pretty much bunkering down...
Just buy a 2nd NFO server (like we did) and move your stuff... it forced us to go to Dallas.
What are you hosting that someone in NYC will care about? I'm thinking people close who wanted low latency are pretty much bunkering down...
Just buy a 2nd NFO server (like we did) and move your stuff... it forced us to go to Dallas.
Re: NYC Datacenter and What it means to customers
anakaoka wrote:while I agree that InterNap should have put fuel pumps higher , maybe even a smaller tank higher up... s*#t happens right?
What are you hosting that someone in NYC will care about? I'm thinking people close who wanted low latency are pretty much bunkering down...
Just buy a 2nd NFO server (like we did) and move your stuff... it forced us to go to Dallas.
I'm sorry but I can't argee with you.
1. The main point of a Datacenter is to stay online when the sh!t hits
2. I already told you I don't got the money (unless I don't want my family to have Christmas) for a 2nd VDS. And why the hell would I pay for a 2nd VDS ?, if NFO wants to give me a 2nd VDS till my NY one back then fine, but I'm not going to pay double for a product.
3. NY is the closes Datacenter NFO has for me, I am on the island of Newfoundland. So unless NFO gets Montreal Canada then NY is what I need.
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!
- RainMotorsports
- A semi-regular
- Posts: 21
- Joined: Tue Apr 24, 2012 4:39 am
Re: NYC Datacenter and What it means to customers
Here is the Data Center - https://maps.google.com/maps?f=q&source ... 80&iwloc=AElochai wrote: I know situation, all I'm staying is what happened ? Datacenters are built to take on the unseen. Why didn't this one have common sub-pumps in the gen rooms to pump out any incoming water ?
Elochai
Its a building that was there long before Internap was there and if your going to build anything in NYC that is often the primary choice unless you can demolish and develop the entire lot.
Its also in an area hit very very hard and while a purpose built data center probably would have survived on generator power the networking in that area might not have survived a storm like this anyway.
Re: NYC Datacenter and What it means to customers
Elochai please open a support request, I will get you a temp VDS for a couple days.
- RainMotorsports
- A semi-regular
- Posts: 21
- Joined: Tue Apr 24, 2012 4:39 am
Re: NYC Datacenter and What it means to customers
Also should add to my post about the location of this building.
The fuel pumps would be at tank level. Its is likely between the building code and the pre existing structure these had to be in the basement. The generators are on a higher level and feed off the "reserve" tanks which are in turn kept full by the pumps in the basement with their main supply.
If the tanks had to be in the basement, the pumps belong there with them. Not really a design choice they had. As far as pumping out the basement. You have to have somewhere for the water to go.
The fuel pumps would be at tank level. Its is likely between the building code and the pre existing structure these had to be in the basement. The generators are on a higher level and feed off the "reserve" tanks which are in turn kept full by the pumps in the basement with their main supply.
If the tanks had to be in the basement, the pumps belong there with them. Not really a design choice they had. As far as pumping out the basement. You have to have somewhere for the water to go.
Re: NYC Datacenter and What it means to customers
I will do that when I'm off. It won't be much help to me I don't think, but it should keep my client happy and not wanting a refund for the down time.hiimcody1 wrote:Elochai please open a support request, I will get you a temp VDS for a couple days.
It to bad they didn't have the room to redesign parts of that building to better support the generator issues. I believe if they were running the server would be still up. The thing about the network in New York is that it is up and running still. Water won't hurt the fiber and the points for each router or relay's seem to be still running as well so far.RainMotorsports wrote:Also should add to my post about the location of this building.
The fuel pumps would be at tank level. Its is likely between the building code and the pre existing structure these had to be in the basement. The generators are on a higher level and feed off the "reserve" tanks which are in turn kept full by the pumps in the basement with their main supply.
If the tanks had to be in the basement, the pumps belong there with them. Not really a design choice they had. As far as pumping out the basement. You have to have somewhere for the water to go.
As for pumps I understand that issue with where to pump water, they would of had to do what we do here, our basement is a sub basement (A basement under a basement). We have to pump it up about 2 floors.
Thank you tho as this is the kind of information I was wondering about . I do hope things get better tho for everyone however as the damage to the area is nuts.
Scott AKA Elochai
Let the Clips Fall Down!
Let the Clips Fall Down!