======================================================
29 November - 11.45am
The RAID rebuild is completed and the RAID array is healthy again. We are aware of three clients on this server who have some issues moving forward and we are working with these clients now. Please note most of these issues are complex issues so we ask you for some patience while we work to resolve these for you.
======================================================
29 November - 9.58am
All VMs are now up and the load on the node (whilst elevated caused by the RAID rebuild) is satisfactory. A few clients are reporting issues. One client needed us to manually restart apache, another needed us to manually restart MySQL and another needed a database table repair. Just one server is having major issues now and the MySQL service will not start up and a tech is actively working on that for that particular client.
If you see any issues we are not yet aware of please let us know right away and we will work to put it right
The RAID rebuild is at 71%
======================================================
29 November - 6.14am
Nine servers are now running and those clients who have tickets open have been updated. They are starting up one at a time. Progress is slow as there is a RAID rebuild being undertaken at the same time (see first post in this thread about missing disk). If your server is not on line yet it will not be long. The RAID rebuild is at 26%
======================================================
29 November - 5.36am
The main Node is now back on line and the VPS Containers are starting up one by one. This process should not take very long. Once that happens we still need to replace the 'missing' disk and rebuild the RAID array (see the first post in this announcement) so that will result in higher loads and higher I/O wait time. We will continue to post here on progress.
======================================================
29 November - 4.47am
The fsck indicates that Inodes are currently being fixed now and not showing a progress bar at this time. We will continue updating you on its progress.
======================================================
29 November - 1.07am
The file system check is 55% completed at this time
======================================================
29 November - 12.25am
We are aware of an issue with VPS4 Node. A short while ago our hardware monitoring solution reported a problem with the RAID array. We discovered upon closer inspection that Drive p0 in the array was being reported as ‘missing’ when it clearly was there. This drive needed to be replaced and the array rebuilt. Please note this is a very normal thing and RAID10 arrays have 4 disks (2 pairs) with redundancy in each pair. One disk from a pair can fail without any impact on server data. The disks are hot swappable and most times this is completed without the client even noticing.
We had a ‘remote hands’ technician in the data centre replace the drive. Unfortunately when he did that the server went into read only mode and required a reboot. The server then insisted on an fsck check (file system check) which is currently at 14%. Unfortunately there is nothing we can do to stop this and we need to let it run to completion.
We will update this announcement as soon as we have more information to tell you and we apologise for any inconvenience. This particular server went live in early 2012 and has had 100% continuous uptime. This is why it is needing the fsck at this time.
======================================================
