Saturday 7 January 7pm : As a precaution we are about to swap out p2 drive in the RAID array following p3 drive being swapper earlier in the week. The drive is not failing but it showing signs that it may do in future. We felt it was a good idea to replace this now as preventative maintenance. This update is just a courtesy as we are not expecting any issues or downtime and the rebuild will complete overnight.
=========================================
This issue is now resolved. We believe the 2 outages in the past week were caused by a degraded drive in the RAID array. In theory a degraded drive should not cause a server outage but since the drive was replaced the server has had perfect uptime for a few days now. Thanks for sticking with us on this one.
=========================================
Update 3.03pm : The RAID rebuild is progressing well and is currently at 20%. All VPS servers on this node are up at this time and this is just a courtesy notification to inform you we have replaced the degraded disk and it is rebuilding normally.
=========================================
Update 11.40am : Many of the containers are started now and the rest are starting up. There is a degraded disk on the array on this server that we are also having to replace and the RAID array will rebuild over the next few hours. This may cause elevated load
=========================================
Update 11.13am : The fsck has completed and the main node is up again. We are working now with our system admins to determine the cause of the issue and also when safe to do so bring all containers back on line asap
=========================================
Hello it's Stephen here. I just wanted to come in on this to let you know that once we have this server back we will do everything we can to make sure it remains solid. We have many VPS nodes and Cloud Hypervisors in our network and all are up bar this one - this one server has caused issues in the past week requiring 2 reboots. We are working with the system admins at the data centre to make sure whatever hardware issue or other issue is causing this is resolved.
The server had had rock solid uptime hardware wise until this week and I can assure you we will do all we can to make sure it is stable as we head into 2012
=========================================
Update 10.48am : The fsck is currently at 71%
=========================================
Update 10.31am : The fsck is currently at 60%
=========================================
Update 10.01am : The fsck is currently at 45%
=========================================
VPS2 node is showing an abnormal RAID array meaning a disk is degraded and needs replaced. A reboot was also required and the server is now performing an FSCK file system check which is causing an extended delay in the server coming back on line.
We are activaly monitoring the server now and will update this announcement once all is good again.
It is no consolation but anyone who has been on this node for a long time will know it has had virtually 100% uptime in recent months and we sincerely apologise today for this outage.
