Feburary 05 01:55
All sites should now be back online. The final hurdle laid with a corrupted Cloudlinux update that was causing CageFS to lock sites out of databases. We have fixed this and all sites appear to be loading. If your site is not, please open a support ticket or email support@bigwetfish.co.uk from your registered email address and we will look into it as quickly as possible. We apologise again for the unexpected and extended downtime on server35.
Feburary 05 01:15
All databases are now restored and every test we have performed (eg mysqlcheck -r mysql , mysql> show engines etc) all show no errors. This leads us to believe the databases are fine. We are still getting database errors on the websites though. We are able to parse the index.php file of some sample sites from the server itself. Logs show invalid passwords when accessed via a web browser. This leads us to believe the issue is actually with php-mysql. We're running cPanel easyapache now to hopefully correct this issue. Apologies for this blip. The server has had virtually 100% uptime in the past 14 months. We will update you soon and senior staff are in our Carrickfergus offices working throughout the night on this if needed.
Feburary 04 23:53
We have been restoring the databases and had limited success. On each restore, the MySQL service came back online, served requests and them corrupted the tables again. We called in our senior Level 3 technician to assess the situation and advise us further. They have managed to correct the issue causing the innodb engine corrupting the tables and the restore is now processing again. We apologise again for this unexpected downtime.
Feburary 04 21:48
Unfortunately, it seems the InnoDB tables in the MySQL server have been corrupted beyond recovery. We have backed up the corrupted databases incase something can be salvaged from them later. We have begun restoring the databases from the backup take at 2am last night. This will result in lost database data from today but files at this time should be unaffected by this restore. We apologise for the downtime and likely loss of data. We will keep you informed as more news comes in
Feburary 04 20:52
An issue with the MySQL service on server35 has caused sites to not resolve or show installation pages due to being unable to connect to the MySQL service. Our senior technicians are currently working on the issue and we will update this announcement with more information as it comes in.
