[Resolved] VPS8 OpenVZ Node Reboot Friday Evening
-
Sunday, 6th October, 2013
-
12:06pm
This is a report on why VPS8 OpenVZ Node needed a reboot on Friday around midnight. OpenVZ is a slightly cheaper form of Virtualization that allows you to get an Equal share of CPU and by its very nature there is no way to limit CPU usage like happens on Xen Virtualization.
At midnight on Friday 4 October 2014 our monitoring system detected high load on this VPS Node. We immediately attempted to log into the server but were unable to. There maybe was a 60 second delay from our alert to us trying to log in. We were forced to console to the server and reboot the box. Once an OpenVZ server reboots all Virtual Containers need to recalculate quotas and boot up in sequence. The time from reboot to all containers being back on line was 20 minutes.
We immediately investigated the cause and we discovered one user with a script that was to populate a database that was running out of control. When we switched off the offending VPS load on the server dropped to a very low level. When we turned on the offending server load spiked to 20+.
We have worked with this client and he has agreed to disable this script while he works with his developer to optimize it. The client actually asked us to apologise to any other affected clients and he genuinely felt bad it was his script that cause a server crash. Hopefully the fact it happened at mdnight on a Friday will not have caused too much inconvenience. The client was offered a move to Xen Virtualization where this script could run and max out his allocated cores without affecting other users.
