How to shink or compact a virtual disk

Most virtualization products support thin provisioned virtual disks, that is vDisks that start small but grow as required by the guest OS.

Now these growable vdisks are great for saving space on the host and overcommitting host storage but over time the guest uses the space for various things and creates and deletes data but the vdisk on the host remains at the grown size, i.e. it doesn't shrink by itself.

VirtualBox and VMware products have commands to "compact" or "shrink" the vdisks.

For VirtualBox:


For VMware Server:


But here is the trick. Even if you run those commands against your vdisks you will likely NOT shrink the vdisk very much at all. That is because the way those utilities identify the "unused" space is by looking for blocks of zero data and when an OS deletes a file from its disk it only removes the entry in the file allocation table but the data actually remains on the disk until it is reused/overwritten and is therefor not "zero" and cannot be reclaimed by the vdisk utilities.

To overcome this I used to run the following script on the guest before running the vdisk utilities.



But the problem with that is that it does actually fill the disk and grows the vdisk to it maximum which means that you must have enough free space on the host to accommodate the full size vdisk before you run the vdisk utilities to compact/shrink it down.

So I recently found a new utility that actually zero fills only unused non-zero blocks. zerofree

zerofree supports ext2, ext3 and ext4 filesystems.

Thankfully this utility has been packaged up by the good people at EPEL for RHEL and CentOS, it is also available in the debian/ubuntu repos.

To install from EPEL


The only catch w/ zerofree is that it must be run against an unmounted or at least read-only volume/disk. For non-root disks this is not a real problem, you can just schedule maintenance, stop your applications/services that are using the volume and umount it before running zerofree against the device.

For example:


But what about root?
You can boot into single user mode or telinit 1 for the same but that will disable the network and your remote access. So I made a little patch that allows zerofree to be run on the root disk during boot.

This patch is against CentOS 5 /etc/rc.d/rc.sysinit in package initscripts-8.45.30-3.el5.centos.

Here is the patch


To apply


Once the patch is applied and zerofree installed you can enable zerofree at boot with


During the next boot zerofree will be run before root is remounted read-write.

Whichever method you choose to zero out unused space within your guest OS you can then reclaim the space on the host.

For example with VirtualBox


This method of zeroing unused space can also be used with SAN solutions that support thin provisioning.

I use DataCore SANmelody thin provisioning HA SAN's and use this method to reclaim SAU's!


Thats it.

VirtualBox init.d script for CentOS

Here is an init.d script for VirtualBox.

This has been designed for and only tested on CentOS 5.5 and VirtualBox 4.0.2. It'll probably work with RHEL and maybe even Debian/Ubuntu but that is untested.

Define which user will run the VMs

echo "VBOX_USER=vbox" > /etc/sysconfig/vboxcontrol

Enable some virtual machines.

mkdir -p /etc/vbox/machines_enabled.d
touch /etc/vbox/machines_enabled.d/vboxtest
touch /etc/vbox/machines_enabled.d/ftp.example.com

Create the vboxcontrol script.

touch /etc/rc.d/init.d/vboxcontrol
chmod 750 /etc/rc.d/init.d/vboxcontrol

Paste the following script into /etc/rc.d/init.d/vboxcontrol

#!/bin/bash

# $Id: vboxcontrol-init.sh 6347 2011-02-13 23:50:24Z$

# chkconfig: 235 98 02
# description: VirtualBox VM service
#
### BEGIN INIT INFO
# Provides:       vboxcontrol
# Required-Start: vboxdrv
# Required-Stop:  vboxdrv
# Default-Start:  2 3 5
# Default-Stop:   0 6
# Description:    VirtualBox VM service
### END INIT INFO

PATH=$PATH:/bin:/sbin:/usr/sbin

[ -f /etc/sysconfig/vboxcontrol ] && . /etc/sysconfig/vboxcontrol
lockfile=${LOCKFILE:-/var/lock/subsys/vboxcontrol}
pidfile=${PIDFILE:-/var/run/vboxcontrol.pid}
RETVAL=0

# --- VBOX_USER must be defined
[ -z "$VBOX_USER" ] && { echo "ERROR: VBOX_USER must be set in /etc/sysconfig/vboxcontrol"; exit 1; }

SU="su $VBOX_USER -c"
VBOXMANAGE="VBoxManage --nologo"
VBOX_LIST_D=${VBOX_LIST_D:-"/etc/vbox/machines_enabled.d"}
VBOX_BASE=${VBOX_BASE:-"/VirtualMachines"}

# --- maximum loops to wait for all VMs to shutdown, 180 * 5 sec = 15mins
MAX_WAIT=180

# Source function library.
. /etc/rc.d/init.d/functions

vboxdrvrunning() {
    lsmod | grep -q "vboxdrv[^_-]"
}

wait_for_closing_machines() {
  count=0
  while [ ${count} -lt ${MAX_WAIT} ]; do
    RUNNING_MACHINES=`$SU "$VBOXMANAGE list runningvms" | wc -l`
    [ $RUNNING_MACHINES = 0 ] && break
    sleep 5
    count=$((count + 1))
  done
}

start() {
  [ -d "$VBOX_LIST_D" ] || exit 0
  echo -n $"Starting VirtualBox VMs: "
  vboxdrvrunning || {
   failure "VirtualBox kernel module not loaded!"
   exit 0
  }
  success
  [ $RETVAL = 0 ] && touch ${lockfile}
  echo

  for VMFILE in ${VBOX_LIST_D}/* ;do
    [ -e ${VMFILE} ] && VM=`basename ${VMFILE}` || continue
    action $"  Starting VM: ${VM} ..." $SU "$VBOXMANAGE startvm "$VM" --type headless >/dev/null 2>/dev/null"
    RETVAL=$?
  done

  return $RETVAL
}

stop() {
  # NOTE: this stops all running VM's. Not just the ones listed in the $VBOX_LIST_D
  echo -n $"Stopping VirtualBox VMs: "
  vboxdrvrunning || {
   failure "VirtualBox kernel module not loaded!"
   exit 0
  }
  success
  [ $RETVAL = 0 ] && touch ${lockfile}
  echo

  $SU "$VBOXMANAGE list runningvms" | while read VM VMUUID; do
    action $"  Stopping VM: ${VM} ..." $SU "$VBOXMANAGE controlvm "$VM" acpipowerbutton >/dev/null 2>/dev/null"
    RETVAL=$?
  done

  action $"  Waiting for VMs to stop ..." wait_for_closing_machines

  return $RETVAL
}

restart() {
    stop && start
}

status() {
    echo $"Checking for running VirtualBox VMs: "
    enabled_vms=( ${VBOX_LIST_D}/* )
    running_vms=( `$SU "$VBOXMANAGE list runningvms" | awk '{print $1}'` )
    for VM in ${running_vms[@]}; do
      VM=`echo $VM | sed 's/\"//g'`
      ip=`$SU "$VBOXMANAGE guestproperty get "${VM}" /VirtualBox/GuestInfo/Net/0/V4/IP" | sed "s/^Value: //"`
      echo $"  ${VM} is running [ip: ${ip}]"
      for (( i = 0 ; i < ${#enabled_vms[@]} ; i++ )); do
        [ -e ${enabled_vms[$i]} ] || continue
        vmname=`basename ${enabled_vms[$i]}`
        if [ $vmname == $VM ]; then
          unset enabled_vms[$i]
        fi
        unset vmname
      done
    done
    for (( i = 0 ; i < ${#enabled_vms[@]} ; i++ )); do
      [ -e ${enabled_vms[$i]} ] || continue
      vmname=`basename ${enabled_vms[$i]}`
      echo $"  ${vmname} is NOT running"
      RETVAL=3
    done
}

startvm(){
  vmname=$1
  if [ -z "$vmname" ]; then
    echo "ERROR: You need to provide a vm name."
    exit 1
  fi
  
  action $"Sending start command: ${vmname} ..." $SU "$VBOXMANAGE startvm "$vmname" --type headless >/dev/null 2>/dev/null"
  RETVAL=$?
  
  return $RETVAL
}

deletevm(){
  vmname=$1
  if [ -z "$vmname" ]; then
    echo "ERROR: You need to provide a vm name."
    exit 1
  fi

  if [ ! -e "${VBOX_BASE}/${vmname}/${vmname}.vbox" ]; then
    echo "ERROR: The VM configuration file could not be found."
    exit 1
  fi
  
  action $"Sending unregistervm command: ${vmname} ..." $SU "$VBOXMANAGE unregistervm "$vmname" --delete >/dev/null 2>/dev/null"
  RETVAL=$?
  
  if [ "${RETVAL}" == 0 ]; then
    [ -e "${VBOX_BASE}/${vmname}" ] && action $"Removing VM files from ${VBOX_BASE}/${vmname} ..." rm -rf "${VBOX_BASE}/${vmname}"
  fi
  
  return $RETVAL
}

controlvm(){
  command=$1
  vmname=$2
  if [ -z "$vmname" ]; then
    echo "ERROR: You need to provide a vm name."
    exit 1
  fi
  
  action $"Sending ${command} command: ${vmname} ..." $SU "$VBOXMANAGE controlvm "$vmname" ${command} >/dev/null 2>/dev/null"
  RETVAL=$?
  
  return $RETVAL
}

case "$1" in
start)
    start
    ;;
stop)
    stop
    ;;
restart)
    restart
    ;;
force-reload)
    restart
    ;;
status)
    status
    ;;
shutdownvm)
    shift
    controlvm acpipowerbutton $@
    ;;
startvm)
    shift
    startvm $@
    ;;
poweroffvm)
    shift
    controlvm poweroff $@
    ;;
resetvm)
    shift
    controlvm reset $@
    ;;
deletevm)
    shift
    deletevm $@
    ;;
*)
    echo "Usage: $0 {start|stop|restart|status|shutdownvm|startvm|poweroffvm|resetvm} [vmname]"
    exit 1
esac

exit $RETVAL

UPDATE: Up-to-date version of this script can always be found https://github.com/abnormaliti/puppet/blob/master/virtualbox/files/vboxcontrol-init.sh

Add the service and enable it.

/sbin/chkconfig --add vboxcontrol && /sbin/chkconfig vboxcontrol on

The service is now installed and enabled. It will start all enabled virtual machines at boot and attempt to shutdown all VMs during the host shutdown.

The script can also be used to control your VMs.

# --- start all enabled VMs
/etc/init.d/vboxcontrol start

# --- stop all VMs, not just enabled ones.
/etc/init.d/vboxcontrol stop

# --- get the status of all VMs
/etc/init.d/vboxcontrol status

# --- start a particular VM
/etc/init.d/vboxcontrol startvm VMname

# --- shutdown a particular VM
/etc/init.d/vboxcontrol shutdownvm VMname

# --- poweroff a particular VM
/etc/init.d/vboxcontrol poweroffvm VMname

# --- reset a particular VM
/etc/init.d/vboxcontrol resetvm VMname

Minimal CentOS installations

Most of my linux servers use CentOS, centos.org. For both bare metal and virtual installations I like to start with the minimum and only install and configure what is needed.

To do the minimalistic install I use kickstart, in fact I use a perl cgi script I wrote many years ago, before Cobbler, to dynamically generate the kickstart files based on templates and options passed to the cgi.

Here is a sample of the resulting kickstart intended for a VM host.

IMPORTANT: This kickstart WILL partition and format the harddisks.
IMPORTANT: This has only been tested with CentOS/RHEL 5. The partitioning will probably not work with CentOS/RHEL 6.

You will need to modify this to your liking.
- repo.example.com: needs to be replaced w/ real centos repos
- custom repo: this is my own internal repo, you can remove this
- hostname: change to your liking
- root password: insert an encrypted password
- authconfig: modify ldap or replace w/ "authconfig --enableshadow --enablemd5"
- puppet: define your puppetmaster or remove
- time: define your time server or remove

# Kickstart file for vbox1.example.com
#
# $Id: default.ks.tpl 6307 2011-02-08 06:06:30Z $

install
reboot
text

firstboot --disabled


url --url http://repo.example.com/linux/centos/5/os/i386/
repo --name=updates --baseurl=http://repo.example.com/linux/centos/5/updates/i386/
repo --name=extras  --baseurl=http://repo.example.com/linux/centos/5/extras/i386/
repo --name=custom  --baseurl=http://repo.example.com/linux/centos/custom/5/i386/
repo --name=epel    --baseurl=http://repo.example.com/linux/centos/epel/5/i386/
repo --name=epel-testing    --baseurl=http://repo.example.com/linux/centos/epel-testing/5/i386/


lang en_US.UTF-8
keyboard us

network --device eth0 --bootproto=dhcp --hostname vbox1.example.com

rootpw --iscrypted $1$lKmKQMA8$1OiJsc8PGoxQsKQ/GM/Hp0

firewall --disabled
authconfig --enableshadow --enablemd5 --enableldap --enableldapauth --ldapserver=ldap.example.com --ldapbasedn="o=People" --enablecache

selinux --disabled

timezone Australia/Melbourne

%include /tmp/part-include

%packages --ignoremissing --nobase
coreutils
yum
rpm
dhclient
wget
e2fsprogs
lvm2
grub
sysstat
redhat-lsb
sendmail
openssh-server
openssh-clients
comps-extras
cracklib-dicts
gnome-mime-data
rmt
tzdata
nss_ldap
ntp
net-snmp
vim-enhanced
xinetd
iptraf
ruby
xorg-x11-xauth
sysstat
puppet
facter
ruby-shadow
augeas
ruby-augeas
# --- needed to see vmware/virtualbox with facter/puppet
dmidecode
# --- needed to "shutdown" with power button or VBoxManage
acpid
-NetworkManager
-system-config-printer
-pcsc-lite
# removed from "base"
-autofs
-anacron
-bluez-utils
-ccid
-ifd-egate
-coolkey
-brltty
-hfsutils
-ibutils
-ipsec-tools
-mcelog
-mtr
-pcmciautils
-rp-pppoe
-smartmontools
-ypbind
# removed from "core"
-Deployment_Guide-en-US

%pre

#!/bin/sh

set $(list-harddrives)

#    $1 = 1st disk name
#    $2 = 1st disk size
#    $3 = 2nd disk name
#    $4 = 2nd disk size
#    so on

let numhd=$#/2

drive1=$1
drive2=$3

# calculate swap
mem=$(grep MemTotal /proc/meminfo | awk '{print $2}')
swap=$(( $mem / 1000 * 2 ))

# max swap should be 3GB
if [ $swap -gt 3000 ]
then
swap=3000
fi

#Write out partition scheme based on whether there are 1 or 2 hard drives

if [ $numhd == "2" ] ; then
#2 drives
echo "# partitioning scheme generated in %pre for 2 drives"                > /tmp/part-include
echo "bootloader --location=mbr --driveorder=$drive1,$drive2"              >> /tmp/part-include
echo "clearpart --all --initlabel"                                         >> /tmp/part-include
echo "part raid.11 --size=100 --ondisk=$drive1"                            >> /tmp/part-include
echo "part raid.21 --size=100 --ondisk=$drive2"                            >> /tmp/part-include
echo "part raid.13 --size=100 --ondisk=$drive1 --grow"                     >> /tmp/part-include
echo "part raid.23 --size=100 --ondisk=$drive2 --grow"                     >> /tmp/part-include

echo "raid /boot --fstype ext3 --level=RAID1 --device=md0 raid.11 raid.21" >> /tmp/part-include
echo "raid pv.01 --level=RAID1 --device=md2 raid.13 raid.23"               >> /tmp/part-include

echo "volgroup vg0 pv.01"                                                  >> /tmp/part-include

echo "logvol swap --fstype swap --name=swap --vgname=vg0 --size=$swap"     >> /tmp/part-include
echo "logvol / --fstype ext3 --name=root --vgname=vg0 --size=300 --grow"   >> /tmp/part-include
else
#1 drive
echo "# partitioning scheme generated in %pre for 1 drives"                > /tmp/part-include
echo "bootloader --location=mbr --driveorder=$drive1"                      >> /tmp/part-include
echo "clearpart --all --initlabel"                                         >> /tmp/part-include
echo "part /boot --fstype ext3 --size=100   --ondisk=$drive1"              >> /tmp/part-include
echo "part swap  --fstype swap --size=$swap --ondisk=$drive1"              >> /tmp/part-include
echo "part /     --fstype ext3 --size=300   --ondisk=$drive1  --grow"      >> /tmp/part-include
fi

[ -f /tmp/part-include ] || touch /tmp/part-include

%post

hostname vbox1.example.com

echo "
PUPPET_SERVER=puppet.example.com
PUPPET_LOG=/var/log/puppet/puppet.log
PUPPET_EXTRA_OPTS=\"--autoflush --pluginsync\"
" >> /etc/sysconfig/puppet

# --- setup cron to make sure puppet keeps running!
echo "*/10 * * * * root [ -f /etc/sysconfig/puppet ] && . /etc/sysconfig/puppet; if [ \"\$WATCH_PUPPETD\" != \"no\" -a -f /etc/init.d/puppet ]; then /sbin/pidof -x puppetd >/dev/null || /sbin/service puppet restart; fi" > /etc/cron.d/puppetd

echo "time.example.com" > /etc/ntp/step-tickers

/usr/sbin/ntpdate -s -b time.example.com

/sbin/chkconfig ntpd on

/sbin/chkconfig puppet on

/sbin/chkconfig network on

# --- disable the default repo's
for xx in /etc/yum.repos.d/*
do
echo "disabling repo file: `basename $xx`"
mv -f ${xx} ${xx}.disabled
done

Replace VMware Server with VirtualBox

For many years I have been using VMware Server 1 and 2 to deploy workgroup servers but it seems VMware have abandoned this product in favor of ESXi. ESXi is a great product and I use that too in my datacenters but for workgroup hosts that do not meet the CPU or hardware requirements the baremetal ESXi is not an option.

So in steps Oracle VirtualBox, www.virtualbox.org. I have been watching VirtualBox for a longtime and some of my colleagues and friends have been using it on their desktops, while I have stuck with VMware Workstation. With the release of version 4 I thought it may be the time to try it out as a replacement for VMware Server.

VirtualBox is primarily intended for desktop use but does have a "headless" mode which could be used in my environment. No desktops on servers here thanks!

All my CentOS 5 based VM hosts start off with a minimal kickstart install of around 800MB and get fully configured by puppet thereafter. A fully installed VM host with CentOS 5.5 and VMware Server 2 fully configured and running is 2G plus virtual machines.

So I will see how VirtualBox compares with VMware Server in installation, size, management and performance.