Ubuntu upgrade can be a challenge
Ubuntu 18 is coming up in a few months. I started using Ubuntu on my machine since version 9.10 and it has gone through 8 upgrades. I’m running 17.10 now. If memory serves me right, my system crashed 2-3 times during these upgrades. In fact, those were the only downtime I have on this machine. In the early days, upgrade isn’t very mature IMO. A system could be partially upgraded and then blocked by dependencies. Since the introduction of do-release-upgrade, the upgrade process is much more reliable. However, it’s still a risky operation when upgrading to a development release. On the other hand, jumping from LTS to LTS is quite safe.
Last night, I tried to upgrade to 18.04 and it was a disaster. xorg refused to start and the system was inoperable. There is no way to downgrade to 17. As a result, I had to restore my system from backup. That took me a bit of time to do. There is a better solution than traditional file backup-restore, AKA LVM snapshot.
Migrate existing root filesystem to LVM
To use LVM snapshot, I first need to migrate the partition containing the root filesystem to a LV. The high-level steps are:
- Close out as many applications as I can
- Create a LV with matching size
- dd the partition to the LV
- Mount the LV on /mnt/newroot
- Bind mount /run /sys /proc /dev on /mnt/newroot
- chroot /mnt/newroot
- Edit /etc/fstab so / is mounted from the LV
- Edit /boot/grub/grub.cfg, change root=/dev/vg?/lv?
- Edit grub.cfg and add insmod lvm
- Update the initramfs image
- Reboot
Let’s do a refresh course on LVM snapshot
LVM snapshot freezes a LV at a point-in-time. On the backend, pnce a snapshot is taken, all future changes are stored in the snapshot volume. However from user point of view, it is kind of reversed. The original volume will continue to be writable, while the snapshot volume is usually used for reading files from the frozen point-in-time. The snapshot volume can also be written to. But for this exercise, that feature is not useful.
At a later time, one can choose to remove the snapshot, or merge it with the origin. When a snapshot is removed, that point-in-time copy of the filesystem is removed. All changes made since the snapshot are applied to the origin. This can be performed online. When a snapshot is merged, we’re actually discarding all the changes made since the snapshot was performed, and fall back the origin volume to that time. A merge operation can only be performed when the origin volume is unmounted. If it’s the root filesystem, then it will happen the next time the volume is activated.
Take a LVM snapshot before upgrade
Now that we have a good understanding of LVM snapshot, we are ready to take one. LVM snapshot must reside in the same VG as the origin volume. One must have free extends in the VG to create a snapshot volume. On my computer, I have a 50G root filesystem (/dev/ssd/rootfs). To create a 30G snapshot volume, run:
lvcreate -L30G -s -n rootfs-snap /dev/ssd/rootfs
Bare in mind the snapshot volume should be big enough to hold all the changes, in this case the delta between 17.10 and 18.04. It probably takes a lot less than 30G, but let’s be generous.
Once a snapshot is taken, start the upgrade. If everything works fine, we can remove the snapshot.
lvremove -f /dev/ssd/rootfs-snap
If things goes south, reboot the computer with a live cd or usb stick. Make sure the root filesystem is not mounted, then issue the merge command
lvconvert --merge /dev/ssd/rootfs-snap
When it’s finished, the root filessystem will be reverted back to the state before the OS upgrade. The snapshot volume will be deleted automatically. I’ve never tested this part and hopefully I never will be. If I do I’ll be sure to update this page. As a prove of concept, I’ve tested snapshot merge on a test filesystem.
LVM snapshot merge (fall back)
Here I’ll demonstrate how LVM snapshot merge works. I have a filesystem called /dev/ssd/vault with 2 files on it:
# ls -lh /mnt/vault total 15M -rw------- 1 root root 7.6M Mar 4 02:04 vmlinuz-4.13.0-36-generic -rw-r----- 1 root root 7.5M Mar 4 02:04 vmlinuz-4.15.7-pos
At t1, I created a snapshot called /dev/ssd/vault-snap
# lvcreate -L2G -s -n vault-snap /dev/ssd/vault Using default stripesize 64.00 KiB. Logical volume "vault-snap" created.
I then created new files on the filesystem. There are at this point 4 files:
# ls -lh /mnt/vault total 1.1G -rw-r----- 1 root root 1.0G Mar 4 02:08 1G.dat -rw-r----- 1 root root 4 Mar 4 02:07 new-write-after-snapshot.txt -rw------- 1 root root 7.6M Mar 4 02:04 vmlinuz-4.13.0-36-generic -rw-r----- 1 root root 7.5M Mar 4 02:04 vmlinuz-4.15.7-pos
lvs also tells me the snapshot volume is 50% used. So the new files appears to be on the original filesystem, but the actual extends or blocks that stores the new files is the snapshot volume.
# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert vault ssd owi-aos--- 10.00g vault-snap ssd swi-a-s--- 2.00g vault 50.20
To revert to the original state, unmount the filesystem, issue lvconvert –merge
# umount /mnt/vault # lvconvert --merge ssd/vault-snap Merging of volume ssd/vault-snap started. vault: Merged: 51.13% vault: Merged: 93.73% vault: Merged: 100.00%
Mount the filesystem again and the files created after t1 are gone:
# mount /dev/ssd/vault /mnt/vault # ls -lh /mnt/vault total 15M -rw------- 1 root root 7.6M Mar 4 02:04 vmlinuz-4.13.0-36-generic -rw-r----- 1 root root 7.5M Mar 4 02:04 vmlinuz-4.15.7-pos
Upgrade to Ubuntu 20.04 Focal
I’ve used these steps once and now let me run it again and upgrade to Ubuntu 20.04. With all the time working from home, this is a critical time, I need my computer to be 100% after the upgrade.
lvcreate -L30G -s -n rootfs-snap /dev/ssd/rootfs do-release-upgrade -d
Glad that the upgrade went well. Upgrade from 19.10 to 20.04 is painless, touchwood. Now that I’m happy, I can remove the snapshot. The only issue so far – google chrome is missing from dash and cannot be added as dash favorite. Starting it with command line is perfectly fine.
lvremove -f /dev/ssd/rootfs-snap
