Data Center Fire - Voicemail

[VOIPoTim]

Hey Guys,

We've been informed that 1 of the Houston The Planet datacenters has had a fire. By order of the fire department, the entire facility is offline. No servers were damaged, but all power is off until the fire department gives the OK and completes their investigation.

One of our voicemail servers was the only server there. They keep messages in sync so nothing should technically be offline, but you may notice minor issues with voicemail because honestly this is the first time we've had a server in the group offline. Everything should take care of itself, but I wanted to let you know in case you do see any issues.

The Planet has no ETA for us at this time on the facility damage. They only could confirm no servers were damaged and dry supression was used. I'm told there is a possibility of significant damage to some of their power equipment though.

Again, this only affects 1 voicemail server and everything should already be routed around it. Their other 5 datacenters were unimpacted.

[chpalmer]

My 19 line vm is working but 16 stops ringing and starts again never going to vm.

Just in case its related.

[VOIPoTim]

My 19 line is working but 16 stops ringing and starts again.

Just in case its related.

On Zeus or Codeblue?

[chpalmer]

19 uses the Grandstream.

16 I have on zeus.

I forget about your post I just read 30 minutes ago about possible issues.


Off to take my memory pills now...

[VOIPoTim]

19 uses the Grandstream.

16 I have on zeus.

I forget about your post I just read 30 minutes ago about possible issues.


Off to take my memory pills now...

No problem.

As far as Zeus, we have a small checklist of things we're working to resolve before it's officially in use, but nothing too major. Anything you notice, please open a ticket.

Next week doing the final round of extension testing and fixing anything that crops up.

[NY Tel Guy]

Hey Guys,

We've been informed that 1 of the Houston The Planet datacenters has had a fire. By order of the fire department, the entire facility is offline. No servers were damaged, but all power is off until the fire department gives the OK and completes their investigation.

One of our voicemail servers was the only server there. They keep messages in sync so nothing should technically be offline, but you may notice minor issues with voicemail because honestly this is the first time we've had a server in the group offline. Everything should take care of itself, but I wanted to let you know in case you do see any issues.

The Planet has no ETA for us at this time on the facility damage. They only could confirm no servers were damaged and dry supression was used. I'm told there is a possibility of significant damage to some of their power equipment though.

Again, this only affects 1 voicemail server and everything should already be routed around it. Their other 5 datacenters were unimpacted.

Wow,
I read the Planet Forum and they said it affected 7,500 customers.
It is also featured on the DSL Reports front page.

[VOIPoTim]

Wow,
I read the Planet Forum and they said it affected 7,500 customers.
It is also featured on the DSL Reports front page.

Yeah, fortunately for us, we only had a small batch of servers there. We originally thought we had about 1K there, but turns out only 186 of them were in use. They're mostly dedicated server customers.

We have about 8,000 servers in their Dallas facilities. The Planet has about 40 full-time people servicing our account and they are all in Dallas and we prefer the Dallas DCs overall so that kind of saved us from putting a lot of stuff in Houston.

They moved m of our servers that were in the impacted DC (I guess when you have almost 10K total and add 500-1000 a month it certainly helps speed them up) to the other Houston DC last night so we're basically all set. Unfortunately, the facility isn't back online yet for all of the others and they may not have 100% of it online until Tuesday now.

We're issuing 100% refunds for 1 month of service to the dedicated customers we have that are impacted.

Something no one can really plan for, but from our discussions with them, they do have everyone working around the clock and are trying to do the best anyone can in this kind of situation.