VPS9 Node File System Errors
-
Saturday, 24th May, 2014
-
21:48pm
========================================================
11.15pm
The node itself is back on line and the fsck completed. There are some files that appear to be corrupt (we do have a complete node backup with our Idera R1Soft system). Servers are starting up now though and we will test each one. So far 3 servers are back up and are fine. Others are starting one at a time.
Any servers that will not start will need restored and we will work on that.
The file system check is completed now but the RAID rebuild is at 20%. it is set to low priority to avoidnexcessive load
========================================================
10.15pm
The server is offline and a manual fsck is running. It is showing 20% completed at this time.
One drive had completely dropped off the array and another was showing errors. These drives are from different pairs in the RAID1 so we are confident this server will come back on line. After the fsck (file system checks) completes we need to rebuild the array.
Unfortunately it is best to do this work with the server in recovery mode to avoid delays caused by I/O when the containers are up. We do apologise for this downtime. This server has had 100% uptime in the past 12 months.
========================================================
Date: 24 May 2014
Time: 10pm
Unfortunately during a routine RAID rebuild on VPS9 (where a drive dropped off the array and was replaced) the file system is showing a significant number of errors.
We are taking the sevrer down for maintenance. We anticipate a 4-6 hour window for this maintenance and we apologise in advance for this unplanned downtime.
Updates will be posted here
