Save State of VirtualBox Virtual Machines on APC UPS power loss shutdown by apcupsd Linux Daemon

The pets in my virtual machine zoo (SANYALnet Labs) lived inside two Dell PowerEdge blades (a 2950 with 8 Xeon cores and 16 threads + a R710 with 12 Xeon cores 24 threads) for over a decade. Finally deciding to do something about the resultant electric bills, I hit eBay and eventually found a Gigabyte H410M motherboard along with an interesting processor: 8-core 16-thread Intel Intel(R) Core(TM) i7-10700T. The "T" translates to "low power". Intel's claims stands up to measurement. The i7-10700T indeed draws about 35 watts at light loads and runs at 105 watts sustained at 85% or more processor usage when all the virtual machines are fired up. The new build with all virtual machines running under Oracle Virtualbox now draws a total of around 150 watts.

This new low power usage means I have now many minutes of battery backup from the connected APC UPS, enough to actually save the states of all the virtual machines before powering the physical machine down.

The apcupsd daemon has been around for ages and is well documented all over the internet. I followed the path of others and with a little bit of tweaking got what I wanred. When the APC UPS signals the system is on battery power, a script is executed that takes a snapshot and saves the state of all the running virtual machines. Other VMs not in running state are not affected.

The Gigabyte H410 motherboard with Intel Core i7-10700T processor and 32GB of RAM runs VirtualBox virtual appliances under Ubuntu 20.04 LTS (5.4.0-80-generic kernel at the time of writing). apcupsd - the APC UPS software for Linux is available from the Ubuntu repositories and easily installed using "apt-get install apcupsd". The version running at the time of writing is "3.14.14-3build1". Here is what "apcaccess status" tells me about my UPS (including incorrect Load % and Timeleft which I don't really care about as long as enough time is left to take snapshots of and save states of all the hosted virtual machine appliances):

I learned by skimming through the man pages that the apcupsd daemon invokes a "/etc/apcupsd/apccontrol" script which handles specific UPS events. On AC mains power fail, the events raised in order are (example of actual sequence with timestamps):

 On closer observation, I found that a new instance of apccontrol script is started off every time an event is sensed by apcupsd, even if prior invocations of apccontrol have not completed. Bummer! This means that if we write a script to start to take snapshots and save current states of hosted running virtual appliances when the "power failure" event occurs, the "system shutdown" event will initiate an immediate shutdown 11 seconds later even if the virtual machines have not been saved yet!

Fortunately, the solution was easy. apccontrol executes scripts with the same name as the events placed in the /etc/apcupsd directory. The event names are listed neatly in a case statement in apccontrol. So, first I created a script called "/etc/apcupsd/do-nothing-event-handler.sh" :

Then, I deleted all existing event handling scripts except "onbattery" and added symbolic links to do-nothing-event-handler.sh. The remaining "onbattery" script is the only one I use to save snapshots and states of the virtual machines and then shut down the machine. Here is what my /etc/apcupsd directory now looks like:

Here is the "onbattery" script that does what I need on AC mains powerfail and switchover to UPS battery power.

The /root/clear-kernel-memory-cache.sh script invoked from "onbattery" reclaims unused memory from the cache. I do this in the hope of maximizing memory and avoiding too much disk activity during the process of taking snapshots and saving states of the the running virtual machines. 

and finally, here is the save-vms-on-ups-power.sh script that is invoked from onbattery to identify the running virtual machines, take snapshots with suitable comments, and save the states. This script is a tweak on the wonderful script I found at ubuntuforums.org.