Server node, Xen31 is down.
-
Wednesday, 2nd September, 2015
-
15:04pm
3 September 2015: 1.14am
The server remained on line after the earlier reboot. We replaced the failed SSD drive from the array at 12.30am and that drive has now sucessfully built onto the array and the array is now optimal.
We thank you for you patience and understanding on this matter. Total downtime was approx 30 minutes around 3.30pm.
2 September 2015: 5:47pmThe server has been back online since 3:36pm this afternoon after a restart successfully got the server out of read-only. We have found that an SSD disk drive in the array needs replace and have scheduled a time to do so. The server node may go down briefly at 12am to replace the drive and then the rebuild will begin at low priority. The rebuild will not affect server node operations.
We thank you for you patience and understanding on this matter2 September 2015: 3.30pm
We have requested a KVM remote connection to the server. The data centre anticipate this will be ready at 3.42pm. As soon as we have access we can begin troubleshooting. As it stands it appears one of the drives has dropped off the array:
Logical device segment information
--------------------------------------------------------
Group 0, Segment 0 : Present (Controller:1,Connector:0,Device:0) 14330CEEDD9A
Group 0, Segment 1 : Present (Controller:1,Connector:0,Device:1) 14350D123FB3
Group 1, Segment 0 : Present (Controller:1,Connector:0,Device:2) 14350D12F6BB
Group 1, Segment 1 : Missing
We will have more information soon.
==============================================
2 September 2015: 3.15pmUnfortunately Xen31 went unresponsive. It appears one of the drives has 'fallen' out of the array and the node has tried to rebuild onto the failed drive causing high Input/Output load. We stopped the array rebuild and load has reduced however the file-system has now gone read-only.We apologise for the trouble and disturbtion this is likely causing and we are working to get the issue resolved as quickly as possible and please stand by for more information.
