LVM Compression and Deduplication

With LVM / VDO, it is possible to save disk space using block-level compression and deduplication. I’ll show you how it’s done on Linux.

Virtual Data Optimizer (aka VDO) is feature from Red Hat, which provides block-level deduplication, compression, and thin provisioning for storage. You can learn more about it on Red Hat’s documentation.

Setting up KVDO and vdo tool

On Red Hat variants, we can do it by installing packages from official repo. Once installed, load the kernel module and you are ready to go.

dnf -y install vdo kmod-kvdo vdo-support
modprobe kvdo

On Debian variants, it is quite a bit more complex. Here is how I get them to install on Ubuntu 22.04.

# install kvdo
git clone https://github.com/dm-vdo/kvdo
cd kvdo
make -C /usr/src/linux-headers-`uname -r`/ M=`pwd` modules
make -C /usr/src/linux-headers-`uname -r` M=`pwd` modules_install
depmod -a
modprobe kvdo

# install vdo tool
# check that libdevmapper package is installed first
git clone https://github.com/dm-vdo/vdo
cd vdo
make -j
make install

Building the kernel module automatically with DKMS

cd /usr/src
git clone https://github.com/dm-vdo/kvdo
mv kvdo kvdo-1.1
cd kvdo-1.1
touch dkms.conf

What you need in dkms.conf

PACKAGE_NAME=kvdo
PACKAGE_VERSION=1.1
BUILT_MODULE_NAME[0]=kvdo
BUILT_MODULE_LOCATION[0]="vdo/"
DEST_MODULE_LOCATION[0]="/extra/"
REMAKE_INITRD=yes
AUTOINSTALL="YES"

Version 1.1 is just an arbitrary number. Once you have the source code in the right place and dkms.conf ready, run

dkms build kvdo/1.1

If successful, run dkms status and it should say installed. Next time you update your kernel, the module will be generated automatically by DKMS

dkms status
kvdo/1.1, 6.1.1-pr0, x86_64: installed (WARNING! Diff between built and installed module!)

Create LVM volume

Now let’s go ahead and create a LVM volume. We use the same lvcreate command plus a few optio

ns. Here, I’ll create a 8G volume with a virtual size of 10G. The volume will be created from the vg01 volume group. Next, format the filesystem like you normally would.

[[email protected] ~]# lvcreate --type vdo --name vol1 --size 5G --virtualsize 10G vg01
    The VDO volume can address 2 GB in 1 data slab.
    It can grow to address at most 16 TB of physical storage in 8192 slabs.
    If a larger maximum size might be needed, use bigger slabs.
[[email protected] ~]# mkfs.f2fs /dev/vg01/vol1

Checking deduplication efficiency

To test how effective VDO is, I created a 500G file with random data. I then copy the file to a new file with a different name. Using the vdostats command, I can see that the second file did not increase disk usage at all and the space saving percentage has increased.

[[email protected] ~]# dd if=/dev/urandom of=/data/random1 bs=1M count=500

[[email protected] ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/compress-vpool0-vpool      5.4G      3.7G      1.6G  69%      1%

[[email protected] ~]# cp /data/random1 /data/random2

[[email protected] ~]# vdostats --si
Device                    Size      Used Available Use% Space saving%
/dev/mapper/compress-vpool0-vpool      5.4G      3.7G      1.6G  69%      47%

Wrapping up

The test above may not be common use case. But I also migrated a general purpose filesystem to VDO, and observed some 63% savings.