DIMEnoc Orlando Issues

Monday 30 September 2013 - Final Update

We have received the Reason for Outage and you can read it here:  

http://www.hostdime.com/blog/wp-content/uploads/2013/09/RFO-09.28.13.pdf

===============================
3.40pm Update

Stephen here again from BWF Management.  All services are coming back on line in Orlando Florida. I have checked a number of servers and it does appear servers were rebooted about 1 hour 40 minutes ago.  Those servers just started pinging to the outside world about ten minutes ago.

Once our provider provide a reason for outage we will make that available.

===============================
3.10pm Update

It's Stephen here again from BWF Management.  Hostdime USA last updated twitter 2 hours ago.  We are unable to get any more information from them.

Hostdime Mexico seem to suggest this issue is power related

https://www.facebook.com/hdcastellano/posts/579089032157974

Click the translate button to read in English.  

===============================
2.55pm Update

Stephen here from BWF Management.  Sorry I have not been on top of this I was actually travelling from Derry/Londonderry to Carrickfergus.  I am now in our office to work on this issue with our other staff.

Hostdime phone support just told me the issue is NOT network related.

I asked if it was power related and the engineer could not tell me. 

Once we have information we will pass it on.

===============================
2.30pm Update

We are unsure what the exact issue is but a tweet to another client states this:

"Servers are coming back up at this time. Our team needs to visit each server so it may take some time for full resolution"

We are unsure why with a network issue a technician would need to physically visit each server.  Unfortunately we are somewhat in the dark here and we will post information as we get it.

===============================
1.45pm Update

We are unable to contact our provider via live chat and our ticket remains in the monitor queue.  There have been no further Twitter updates in the past 15 minutes.

You may want to check out @hostdime on Twitter  or add a column for 'hostdime' in Tweetdeck which is where we are getting our information from at this time.

===============================
1.30pm Update

Our provider Vice President of Operations has just tweeted the following

"We have isolated the issue affecting our data center to 16 cabinets. Those are being actively resolved at this time. Thank you ^DH"

This issue is ONLY affecting our small % of servers we have in the USA.  All UK servers both in DIMEnoc Maidstone and IOMART Maidenhead are unaffected

===============================
We are aware of an issue with our USA based servers in Orlando Florida.  We were initially unable to contact our provider via live chat, ticket or phone and we see other clients are tweeting about an outage so it does appear to be a data centre wide issue.

Just a few moments ago we got this update:

"Thank you for contacting us today. We are currently dealing with an internal error on our end. Our System Administrators are aware of the issue and are working diligently to resolve the error at this time. We are going to put this ticket in our Monitor que so we may update you on the progress of resolving this issue. We would like to thank you for your patience at this time. If you have any other questions or concerns, please feel free to contact us."

We are fully aware a similar outage happened on 30 August 2013 and we take uptime seriously.  We regularly review our choice of hosting partners but please know we have been with DIMEnoc in Orlando for ten years and we have built up ten years of trust.  We will post here once we know more.