Update 8.19am
We reinstalled all Easy Apache Packages on the server and rebooted and sites are now loading. Not sure what happened them overnight as sites were loading when the senior team logged off. but we assume this is a result of the corruption yesterday. All seems to be good now and we will continue to monitor.
---------------------------------------
Update 7.30am
The server came back on line last night but this morning there is an error with PHP. We are urgently looking into this. At the same time another staff member is prepping a new server build. We are aware this situation is not ideal and we can only point to the fact this server has had many years of 100% uptime. Once this server is back we will email all clients to urgently move them. We are working as fast as we can.
---------------------------------------
Update 11.07pm
Server is back on line.
---------------------------------------
Update 11.00pm
Server is moved. We are just doing a file system chgeck on the disk now. Thanks for being patient for a little while longer.
---------------------------------------
Update 9.15pm
A client is asking why this is taking so long given this is a server with only a handful of clients. It's an older server but unfortunately has a very large disk volume so the disk copy process on this particular type of VM needs to copy the entire disk allocation as opposed to disk used. The file copy is at 80% at this time. Next update 11pm.
---------------------------------------
Update 8.00pm
The file copy is at 65%. We still believe this is the best option to bring this server back on line with no data loss. Unfortunately the copy is taking a long time. Next update at 10pm.
---------------------------------------
Update 6.00pm
The file copy is at 46%. We still believe this is the best option to bring this server back on line with no data loss. We appreciate your extended patience here. To the remaining clients on this older server we would recommend once it is back on line you reach out to us so we can move you to a newer server woth a modern operating system that is not end of life. Next update at 8pm.
---------------------------------------
Update 4.00pm
We are fairly confident we will get this old server back on line to avoid a restore but it requires moving it to another physical node. The copy is underway and it should take 3-4 hours. If any client cannot wait please contact us and we can manually restore you from last night's backup to another server. However it you can wait we are fairly sure this will work and the sites will come back on line. next update at 6pm.
---------------------------------------
Update at 3.00pm
The server is still being worked on by our senior team and they are confident they can get the old server back on line which is a preferred option. We will make a final decision at 6pm on whather we need to go to a full restore. Again we apologise for this unplanned outage caused by disk corruption. It's not something we could have foreseen and server 801 has had 100% uptime for a number of years.
---------------------------------------
Server 801 is one of our older cPanel servers in London and unfortunately it has gone read only and the disk is corrupt. We are currently working on this.
This is an older CentOS 7 end of life server so we will be restoring this to a new operating system that may cause compatibility issues with older websites. We will need to work with any clients whose sites are not compatible after this is restored.
We will update clients here as we know more. There are just a small handful of affected clients given this is an older server but we do apologise for the outage.
