With LVM / VDO, it is possible to save disk space using block-level compression and deduplication. I’ll show you how it’s done on Linux.
Virtual Data Optimizer (aka VDO) is feature from Red Hat, which provides block-level deduplication, compression, and thin provisioning for storage. You can learn more about it on Red Hat’s documentation.
Setting up KVDO and vdo tool
On Red Hat variants, we can do it by installing packages from official repo. Once installed, load the kernel module and you are ready to go.
dnf -y install vdo kmod-kvdo vdo-support modprobe kvdo
On Debian variants, it is quite a bit more complex. Here is how I get them to install on Ubuntu 22.04.
# install kvdo git clone https://github.com/dm-vdo/kvdo cd kvdo make -C /usr/src/linux-headers-`uname -r`/ M=`pwd` modules make -C /usr/src/linux-headers-`uname -r` M=`pwd` modules_install depmod -a modprobe kvdo # install vdo tool # check that libdevmapper package is installed first git clone https://github.com/dm-vdo/vdo cd vdo make -j make install
Building the kernel module automatically with DKMS
cd /usr/src git clone https://github.com/dm-vdo/kvdo mv kvdo kvdo-1.1 cd kvdo-1.1 touch dkms.conf
What you need in dkms.conf
Version 1.1 is just an arbitrary number. Once you have the source code in the right place and dkms.conf ready, run
dkms build kvdo/1.1
If successful, run dkms status and it should say installed. Next time you update your kernel, the module will be generated automatically by DKMS
kvdo/1.1, 6.1.1-pr0, x86_64: installed (WARNING! Diff between built and installed module!)
Create LVM volume
Now let’s go ahead and create a LVM volume. We use the same lvcreate command plus a few optio
ns. Here, I’ll create a 8G volume with a virtual size of 10G. The volume will be created from the vg01 volume group. Next, format the filesystem like you normally would.
[[email protected] ~]# lvcreate --type vdo --name vol1 --size 5G --virtualsize 10G vg01 The VDO volume can address 2 GB in 1 data slab. It can grow to address at most 16 TB of physical storage in 8192 slabs. If a larger maximum size might be needed, use bigger slabs. [[email protected] ~]# mkfs.f2fs /dev/vg01/vol1
Checking deduplication efficiency
To test how effective VDO is, I created a 500G file with random data. I then copy the file to a new file with a different name. Using the vdostats command, I can see that the second file did not increase disk usage at all and the space saving percentage has increased.
[[email protected] ~]# dd if=/dev/urandom of=/data/random1 bs=1M count=500 [[email protected] ~]# vdostats --si Device Size Used Available Use% Space saving% /dev/mapper/compress-vpool0-vpool 5.4G 3.7G 1.6G 69% 1% [[email protected] ~]# cp /data/random1 /data/random2 [[email protected] ~]# vdostats --si Device Size Used Available Use% Space saving% /dev/mapper/compress-vpool0-vpool 5.4G 3.7G 1.6G 69% 47%
The test above may not be common use case. But I also migrated a general purpose filesystem to VDO, and observed some 63% savings.