Packet Loss Friday Night

=====================================

WE POSTED AN INCORRECT LINK IN THE MESSAGE WE JUST SENT - PLEASE CHECK HERE FOR THE ISSUES ON 29 MAY

http://www.bigwetfish.co.uk/whmcs/announcements/120/UK-Data-Centre-Connectivity-Issues.html

=====================================

If you had packet loss on Friday night we are sorry and this letter that we just sent to all clients with VPS servers on the affected Node will explain why.  Unfortunately on Friday evening this issue affected a number of other servers too as the Outbound DDOS originating from one of our VPS containers was massive and took a while to mitigate.

Good Morning

I wanted to personally send a quick email to all the clients affected by the outage this weekend and explain what happened.  The VPS node you are on houses a number of Virtual Containers and has been working without issue for a number of months with 100% uptime.

On Friday evening at approximately 11pm we started getting reports of packet loss on multiple servers on our network.  The common denominator was all affected servers were located in the same cabinet in the data centre.  We tracked the source of the packet loss down to this VPS node that houses your container.  When we turned off the VPS Node the problem went away.

Our technicians then started to try to determine the cause of the issue and it was tracked down to an outbound DDOS (Denial of Service) originating from one of the Virtual containers on this node.  We then restarted the node and restarted all virtual containers including your server and left the offending server switched off.

We worked to find the cause of the abuse and we thought we had.  Unfortunately on Saturday night at the exact same time the same issue happened.  We immediately shut the node down as the packet loss meant we had no ssh access to shut the offending container down.  We then rebooted the node immediately again whilst keeping the offending container off line.

I want to assure you we take uptime seriously and this incident was unfortunate.  It was a VPS server belonging to another client that had been compromised in some way and was being used to launch an outbound DDOS which was causing the cabinet switch to become saturated with data packets.

We are confident we have found the cause of this issue and it should not occur again.  The offending server was turned on again yesterday and we monitored the node throughout the night and there were no issues.

This should serve as a friendly reminder to keep ALL installed scripts up to date on your VPS.  We found on the offending VPS container a Wordpress Install that was 14 versions out of date.  This install had been compromised most likely due to it being so out of date.  We found a list of IPs in a folder called tmp in the /tmp directory of the offending server that re-appeared when it was deleted.  We suspect these were the Ips being targeted by the outbound DDOS..  We believe this entire issue was caused by one massively out of date Wordpress Install allowing an attacker to gain access to that one VPS and use it to play its part in a denial of service attack against a list of IP addresses.

Please be assured on a VPS server each Container is completely separate so your server has not been compromised in any way.

Please do come back to me if you have any questions and I will be happy to answer them.  I apologise once again for the downtime.

Stephen K

BWF Hosting