Search

Saturday, December 3, 2016

Zero Out Free Disk Space on Virtual Machines and Compact them Before Backup: Solaris, Linux, Windows, OpenVMS

Before taking backups of my hobbyist and production virtual machines, I follow the popular recommendation of zeroing out all unused free virtual hard drive space and compacting them using the virtual disk compacting tool that comes with Oracle Virtualbox.

Download: All the tools, utilities and scripts described in this post are available for direct download from my google drive.

1. Linux


To zero out unused disk space on my virtual hard disks on Linux virtual appliances, I use Ron Yorston's nice "zerofree" tool.

Start off by building zerofree from source and installing it on your Linux VM. The steps are:

# yum -y install e2fsprogs-devel
# wget http://frippery.org/uml/zerofree-1.0.3.tgz
# tar xvzf zerofree-1.0.3.tgz
# cd zerofree-1.0.3
# make
# cp zerofree /usr/sbin/

Here is a screen-log of me executing the above steps in a directory /tmp/x:

[root@sanyalnet-cloud-vps ~]# cd /tmp
[root@sanyalnet-cloud-vps tmp]# mkdir x
[root@sanyalnet-cloud-vps tmp]# cd x
[root@sanyalnet-cloud-vps x]# yum install e2fsprogs-devel -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.globo.tech
 * epel: mirror.math.princeton.edu
 * extras: centos.mirror.globo.tech
 * updates: centos.mirror.netelligent.ca
Resolving Dependencies
--> Running transaction check
---> Package e2fsprogs-devel.x86_64 0:1.42.9-7.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
 Package                                 Arch                           Version                              Repository                    Size
================================================================================================================================================
Installing:
 e2fsprogs-devel                         x86_64                         1.42.9-7.el7                         base                          70 k

Transaction Summary
================================================================================================================================================
Install  1 Package

Total download size: 70 k
Installed size: 161 k
Downloading packages:
e2fsprogs-devel-1.42.9-7.el7.x86_64.rpm                                                                                  |  70 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : e2fsprogs-devel-1.42.9-7.el7.x86_64                                                                                          1/1
  Verifying  : e2fsprogs-devel-1.42.9-7.el7.x86_64                                                                                          1/1

Installed:
  e2fsprogs-devel.x86_64 0:1.42.9-7.el7

Complete!
[root@sanyalnet-cloud-vps x]# wget http://frippery.org/uml/zerofree-1.0.3.tgz
--2016-12-04 00:32:03--  http://frippery.org/uml/zerofree-1.0.3.tgz
Resolving frippery.org (frippery.org)... 93.93.131.127, 2a00:1098:0:86:1000::10
Connecting to frippery.org (frippery.org)|93.93.131.127|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8506 (8.3K) [application/x-gzip]
Saving to: ‘zerofree-1.0.3.tgz’

100%[======================================================================================================>] 8,506       --.-K/s   in 0s

2016-12-04 00:32:03 (64.9 MB/s) - ‘zerofree-1.0.3.tgz’ saved [8506/8506]

[root@sanyalnet-cloud-vps x]# tar -zxf zerofree-1.0.3.tgz
[root@sanyalnet-cloud-vps x]# cd zerofree-1.0.3
[root@sanyalnet-cloud-vps zerofree-1.0.3]# make
gcc  -o zerofree zerofree.c -lext2fs
[root@sanyalnet-cloud-vps zerofree-1.0.3]# ls -lrt
total 44
-rw-r--r-- 1 root  root   17921 Aug 12  2007 COPYING
-rw-rw-r-- 1 root  root    3870 Aug  9  2012 zerofree.c
-rw-rw-r-- 1 root  root     109 Aug  9  2012 Makefile
-rwx------ 1 root  root   13353 Dec  4 00:32 zerofree
[root@sanyalnet-cloud-vps zerofree-1.0.3]# which zerofree
/usr/bin/which: no zerofree in (/sbin:/bin:/usr/sbin:/usr/bin)
[root@sanyalnet-cloud-vps zerofree-1.0.3]# cp zerofree /usr/sbin/
[root@sanyalnet-cloud-vps zerofree-1.0.3]# which zerofree
/sbin/zerofree
[root@sanyalnet-cloud-vps zerofree-1.0.3]#

zerofree works only with inactive partitions that are not mounted (it correctly refuses to manipulate mounted active read-write partitions). We need to boot into single-user mode and dismount the partitions one by one, running zerofree on each.

To boot into single-user mode, I edit the grub boot command line to add a "s" at the end. To do this, I reboot the virtual machine, and when it comes back to the grub menu, I press "e" to edit. Note: I am using a CentOS release 6.8 Linux virtual machine (kernel 2.6.32-642.11.1.el6.x86_64) for this walk-through.

grub boot menu - hit "e" to edit the boot command

On the next screen, I choose the "kernel" line by pressing the down arrow, and press "e" again to edit the kernel boot parameters line.

Grub Kernel Boot Parameters Selection - press "e" ti edit

In the minimal line editor that opens up, I add a "s" at the end of the existing line, after "quiet", for single-user boot.

add "s" at end of grub boot command line to boot into single-user mode

Pressing Enter after adding the "s" at the end brings me back to the boot screen, where I can now press "b" to boot into single-user.

after adding "s" to the boot command line, press "b" to boot into single-user

The computer boots up into single-user, and drops me into a root shell.


single user boot - root shell
I type in the "mount" command to look at the mount-points to identify the partitions to run zerofree on.

single-user root shell - mount points and partitions

In this case, there are two disk drives. The first disk, sda, has the logical volume managed (LVM) ext4 file-system "/dev/mapper/vg_dormarth-lv_root" mounted on "/", and the ext4 partition /dev/sda1 mouted at file-system "/boot" . The second disk, sdb, has the /dev/sdb1 partition mounted as the ext4 file-system at /home/tracks.

Therefore, we will dismount each of the following in turn and run zerofree on each dismounted file-system:

  • /dev/mapper/vg_dormarth-lv_root mounted as /
  • /dev/sda1 mounted as /boot
  • /dev/sdb1 mounted as /home/tracks
We start off by unmounting "/" and executing zerofree with the verbose option on /dev/mapper/vg_dormarth-lv_root, using the following commands:

# umount /
# zerofree -v /dev/mapper/vg_dormarth-lv_root mounted

run zerofree on "/" file-system
Continuing on with the other two partitions for zerofree:

# umount /boot
# zerofree -v /dev/sda1
# umount /home/tracks
# zerofree -v /dev/sdb1

zerofree more linux partitons

We have completed running zerofree on the partitions on this Linux virtual appliance, and halt the VM using the halt command:

# halt


The halt command should power the VM appliance off (if it does not, use VirtualBox to power it off). The next step is to invoke the Virtualbox manager to compact (shrink) the virtual disk as described below in section 4.

2. Solaris


On my Solaris 11 openindiana virtual machine (http://sanyal.duckdns.org:81), I use this script which invokes gnu dd to create big files filled with zeroes and deletes them, leaving zeroes on the virtual hard disk.
I then shutdown and poweroff the Solaris 11 openindiana VM and use Virtualbox manager to compact (shrink) the virtual disk as described below in section 4.

3. Microsoft Windows NT, XP, Vista, 7, 8, 10


To write zeroes to unused disk space on Windows, I use the SDELETE.EXE command line tool available for free from Microsoft. You can download the SDELETE.EXE tool by itself, or grab the entire Sysinternals Suite which is a collection of great utilities.

But before zeroing out unused space on a Windows NTFS file-system, I try to free up as much space as I can and defragment/optimize the NTFS partitions, so that the effect of zeroing out empty space is maximized.

I first run the "Disk Cleanup" tool that comes with Windows, choosing the "Cleanup System Files" option that clears up unneeded files freeing up the maximum space.

Microsoft Windows Disk Cleanup

Then I run Microsoft's "Optimize Drive" tool that is also included in Windows to defragment my NTFS partition. This tool (at least on Windows 10) is intelligent in handling SSD drives if you have any (I do not). It makes multiple passes on each NTFS partition you choose, relocating, defragmenting and consolidating in each pass and takes a while. Make sure you defragment all your virtual NTFS drives (C: and D: in this example).

Microsoft Windows NTFS Disk Defragment and Optimize Drive Tool

Lastly I run the SDELETE utility from a Command shell launched as an Administrator (Start -> Search for CMD.EXE -> Right Click on cmd.exe in search results -> Run as Administrator). I use the "-z" option for "Zero free space (good for virtual disk optimization)" and launch the 64-bit version SDELETE64.EXE included in the downloaded zip file since I am using 64-bit Windows 10. I do this for every NTFS drive on my Windows VM, C: and D: drives in the following example.

C:\Program Files\SDelete>sdelete64.exe

SDelete v2.0 - Secure file delete
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

usage: sdelete [-p passes] [-r] [-s] [-q] <file or directory> [...]
       sdelete [-p passes] [-z|-c [percent free]] <drive letter [...]>
       sdelete [-p passes] [-z|-c] <physical disk number>
   -c         Clean free space. Specify an option amount of space
              to leave free for use by a running system.
   -p         Specifies number of overwrite passes (default is 1)
   -r         Remove Read-Only attribute
   -s         Recurse subdirectories
   -z         Zero free space (good for virtual disk optimization)
   -nobanner  Do not display the startup banner and copyright message.

Disks must not have any volumes in order to be cleaned.


C:\Program Files\SDelete>sdelete64.exe -z c:

SDelete v2.0 - Secure file delete
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - www.sysinternals.com

SDelete is set for 1 pass.
Cleaning free space on C:\: 100%
...

C:\Program Files\SDelete>sdelete64.exe -z d:
...
...


Running SDELETE to zero free unused Windows NTFS disk space

Once SDELETE is complete, shut down your Windows VM appliance and proceed to compacting the virtual disks described below in section 5.


4. MS DOS, PC DOS, Windows 3.1, Windows for Workgroups 3.11


For my MS DOS and DOS-based Windows 3.x virtual machines, I use the classic Norton Utilities SPEEDISK and WIPEINFO applications to zerofree usused disk space. Norton Utilities 8.0 for DOS and Windows 3.1 is available from many abandonware archives including here.

As soon as "Starting DOS" appears on booting up, press F5 to bypass processing of the startup files CONFIG.SYS and AUTOEXEC.BAT and get to directly to the DOS prompt. CD to the directory where Norton Utilities is installed (typically C:\NU) and run SPEEDISK with "Full with File Re-Order" optimization option for directories and files.

Supratim Sanyal's Blog: SPEEDISK for MS DOS FAT Windows 3.1 Windows 3.11 WFW ZEROFREE SDELETE free disk space for compacting virtual hard disk drive
Norton Utilities Speedisk MS DOS / Windows 3.1 / Windows 3.11 Disk Defragmenter

Then Run WIPEINFO to zero out unusued disk space. A note of caution: MAKE SURE "Wipe unused areas only" IS CHECKED; OTHERWISE ALL DATA ON THE DRIVE WILL BE LOST!

Here are a couple of screenshots of WIPEINFO in action.

Supratim Sanyal's Blog: SPEEDISK for MS DOS FAT File System Windows 3.1 Windows 3.11 WFW ZEROFREE SDELETE free disk space for compacting virtual hard disk drive

Supratim Sanyal's Blog: SPEEDISK for MS DOS FAT File System Windows 3.1 Windows 3.11 WFW ZEROFREE SDELETE free disk space for compacting virtual hard disk drive

Supratim Sanyal's Blog: SPEEDISK for MS DOS FAT File System Windows 3.1 Windows 3.11 WFW ZEROFREE SDELETE free disk space for compacting virtual hard disk drive

Supratim Sanyal's Blog: SPEEDISK for MS DOS FAT File System Windows 3.1 Windows 3.11 WFW ZEROFREE SDELETE free disk space for compacting virtual hard disk drive
Norton Utilities WIPEINFO for MD DOS virtual disk FAT file system zero free space


When complete, power the VM off. The virtual hard drive is ready for compaction.

For defragmenting, I also sometimes use the PC Tools 2.0 Optimizer utility from "PC Tools - WinShield 2.0" group installed with PC Tools 2.0 for Windows.

Supratim Sanyal's Blog: Defragmenting with PC Tools for Windows 2.0 Drive Optimizer on Windows for Workgroups WFW 3.11 MS DOS FAT File System
PC Tools for Windows 2.0 Drive Optimizer



5. VirtualBox Compact (Shrink) VDI Virtual Hard Disk


Once the unused space on the virtual hard disks has been zeroed out, Shut down the Virtual Machine, power it off completely, and use the Oracle VirtualBox "vboxmanage" tool with the "modifyhd --compact" command on your VDI format virtual hard disk to compact it.

For example,

/usr/bin/vboxmanage modifyhd --compact /home/VirtualMachines/OpenIndiana-Solaris/OpenIndiana-Solaris.vdi


If your virtual hard disk file is not in VDI format, you can create a copy of the virtual disk in VDI format using the clonehd function of vboxmanage, like:

/usr/bin/vboxmanage clonehd vmdisk.vmdk vmdisk.vdi --format VDI

You can then reconfigure your virtual machine to use the VDI format virtual hard-disk file and not use your old vmdk or other format virtual hard disk file any more.

As an example of what the virtual disk compacting procedure achieves, here are the size of the virtual disks before and after compacting of the Linux CentOS 6 VM we ran zerofree on in section 1. The system disk size came down by 6 Gigabytes, from 20GB to 14GB, not a small number.

BEFORE
[root@anubis-mighty-anubis Minecraft and WBRi Stream Server.x64]# ls -l
total 29354028
-r--r--r--. 1 root root   401604608 Apr 13  2015 CentOS-6.6-x86_64-minimal.iso
drwxr-xr-x. 2 root root          94 Nov 21 22:08 Logs
-rw-r--r--. 1 root root 20602421248 Dec  3 16:31 Minecraft and WBRi Stream Server V2 x64 OS Disk.vdi
drwxr-xr-x. 2 root root           6 May  5  2015 Snapshots
-rw-r--r--. 1 root root  9054453760 Dec  3 16:31 WBRiBroadcastMaterial.vdi
-rw-------  1 root root       10589 Oct  3  2015 WBRi_Stream-Minecraft.x64-1.14-linux.vbox
-rw-------  1 root root       10353 Dec  3 16:31 WBRi_Stream-Minecraft.x64.vbox
-rw-------  1 root root       10353 Nov  9 14:10 WBRi_Stream-Minecraft.x64.vbox-prev

EXECUTE COMPACT COMMANDS
[root@anubis-mighty-anubis Minecraft and WBRi Stream Server.x64]# vboxmanage modifyhd --compact Minecraft\ and\ WBRi\ Stream\ Server\ V2\ x64\ OS\ Disk.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
[root@anubis-mighty-anubis Minecraft and WBRi Stream Server.x64]# vboxmanage modifyhd --compact WBRiBroadcastMaterial.vdi
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%


AFTER
[root@anubis-mighty-anubis Minecraft and WBRi Stream Server.x64]# ls -l
total 23014440
-r--r--r--. 1 root root   401604608 Apr 13  2015 CentOS-6.6-x86_64-minimal.iso
drwxr-xr-x. 2 root root          94 Nov 21 22:08 Logs
-rw-r--r--. 1 root root 14111735808 Dec  3 16:41 Minecraft and WBRi Stream Server V2 x64 OS Disk.vdi
drwxr-xr-x. 2 root root           6 May  5  2015 Snapshots
-rw-r--r--. 1 root root  9053405184 Dec  3 16:43 WBRiBroadcastMaterial.vdi
-rw-------  1 root root       10589 Oct  3  2015 WBRi_Stream-Minecraft.x64-1.14-linux.vbox
-rw-------  1 root root       10353 Dec  3 16:31 WBRi_Stream-Minecraft.x64.vbox
-rw-------  1 root root       10353 Nov  9 14:10 WBRi_Stream-Minecraft.x64.vbox-prev


6. OpenVMS


I run a VAXserver 3900 and a VAX-11/780 using the SIMH simulator on Linux hosts. Both VAXen run OpenVMS VAX 7.3.

There is no virtual disk compaction tool included with SIMH VAX emulator. However, with the goal of minimizing tarball sizes while taking backups of the OpenVMS systems, I wrote and compiled the following tiny C program which creates a file ZEROFILE.ZERO in the current directory, occupying all available free space on the current disk.

Run the program below on every disk drive, deleting ZEROFILE.ZERO when complete on each drive. This created file must be deleted before the virtual VAXen are shutdown and their SIMH Virtual Machine directory etc. backed up.

After zeroing out empty disk space on my OpenVMS VAX disks, I did see some impact on the tarball size when I backed up the virtual SIMH VAX to a tar-gzipped archive.  The archive size went down from 1,149,655,119 bytes to 1,038,719,403 bytes.

If you want to try ZEROFILE.EXE yourself, download the C source and VAX executable from the FAL SERVER area on my VAXserver 3900 node QCOCAL.



7. Other Systems


A generic approach to zeroing out unused disk space on all Unix-like systems is to simply use the /dev/zero device to read from and dump into a file to fill up unused space, and then delete the file. This method also has the advantage of not bringing the virtual machine down, it can be done on a live system.

I use this approach to zero out unused disk space on my Sophos UTM and pfSense virtual machines.


Sophos UTM



Here's output of the script:

anubis-sophos:/root # nice -n 19 ionice -c3 ./zerofree.sh
+ cd /opt/inst/
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'
+ cd /var/storage/
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'
+ cd /var/log
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'
+ cd /tmp
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'
+ cd /
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'
+ cd /var/sec
+ cat /dev/zero
cat: write error: No space left on device
+ rm -fv zero.delete-me
removed `zero.delete-me'


pfSense (freeBSD)


On my pfSense open-source router virtual machine, I run this script:


Here is what I get:

[2.3.2-RELEASE][root@anubis-pfsense.sanyalnet.lan]/root: nice -19 ./zerofree.sh
+ cd /
+ cat /dev/zero

/: write failed, filesystem is full
cat: stdout: No space left on device
+ rm -v zero.delete-me
zero.delete-me


No comments:

Post a Comment

Recommended Products from Amazon