Shared Server 14
-
Sunday, 8th June, 2014
-
12:49pm
As part of our server security maintenance we have been upgrading the linux kernel on all Cloudlinux machines as recommended in an urgent security advisory received overnight. Server 14 has given us some issues and is currently offline. We sincerely apologise for this outage and we will document this issue here.
===================================================
Sunday 8 June : 8.32pm
The server is back on line. The extended outage today was caused by a number of factors including:
- Auto fsck checks failing
- Delays in our data centre getting KVM sessions provisioned
- Delays getting recovery sessions provisioned
- Manual fsck taking a long time to complete
- A 30 minute time frame to have the recovery mode removed and networking restored
If anyone has any more questions please log a helpdesk ticket and a member of our team will be glad to assist you. We are really sorry about this delay today. The provisioning of KVM sessions etc is usually very fast but it appears the data centre staff today were under pressure.
===================================================
Sunday 8 June : 8.16pm
The fsck has completed and the server is now at the login prompt. We need to wait for our data centre partners to enable networking again and remove the server from the recovery LAN. This should not be much longer.
===================================================
Sunday 8 June : 8.04pm
The file system check is now clearing up some inodes and is at the final stage.
===================================================
Sunday 8 June : 7.50pm
The file system check is still running - 82%
===================================================
Sunday 8 June : 7.30pm
The file system check is still running - 56%
===================================================
Sunday 8 June : 7.15pm
The file system check is still running - 40%
===================================================
Sunday 8 June : 6.00pm
The file system check is running
===================================================
Sunday 8 June : 4.32pm
Data centre staff apologise they are very busy which they say is unusual for a Sunday afternoon. The server emergency KVM session had expired so they are connecting another session now. We are then going to boot the server into recovery mode again and perform once final file system check on the unmounted file system (manually). There was a 2 hour delay caused waiting on a data centre tech to perform a task and whilst we are not shifting responsibility we simply wanted to let you know why there was a delay.
===================================================
Sunday 8 June : 3.55pm
Not wanting to shift responsibility as clearly we are responsible for our servers and this is purely to show we are still waiting. There is slight delay caused by our data centre staff - we are waiting a significant amount of time for them to boot the server with LiveCD. We will update you as soon as practically possible and we are deeply sorry for this outage.
===================================================
Sunday 8 June : 3.00pm
We are still having some problems getting this server to boot. We are working on this with top priority and will have another update at 4pm if not before.
===================================================
Sunday 8 June : 1.50pm
Unfortunately the file system check failed. We are booting the server into a recovery session to perform a manual file system check. We are currently waiting on a data centre admin to boot the server into rescue mode. The ETA is 30 minutes for the request. Apologies for this extended outage
===================================================
Sunday 8 June : 1.30pm
The system is performing another file system check and it currently is at 60%. Updates shortly.
===================================================
Sunday 8 June : 12.50pm
Server 14 was being upgraded to a new Kernel (security recommendation). When the server was rebooted into the new Kernel the server forced a file system check. This file system check failed at 90%. A second check completed but the server now has a read only file system. We are working towards a resolution for this now and we thank you for your patience.
