VG unavailable after upgrade from 18.04 to 20.04 Cannot process volume group vg01 Volume group "vg01" not found

[veribaka]

After upgrading a laptop configured with LVM from 18.04 to 20.04 the volume group is no longer found.
During boot the console shows

Volume group "vg01" not found
Cannot process volume group vg01
Gave up waiting for suspend/resume device
ALERT! /dev/mapper/vg01-root does not exist. Dropping to a shell!

Attached dist-upgrade logs and console output.

Tried the "vgck --updatemetadata vg01" recommended on this bug report: https://bugs.launchpad.net/ubuntu-z-systems/+bug/1874381

[Frank Heimes (fheimes)]

I'm changing this ticket to lvm2 and focal, removing the currently marked as affected "Ubuntu on IBM z Systems" project, since the logs show that they come from a amd64 system.
Nevertheless, the referenced LP 1874381, was on s390x.

[Frank Heimes (fheimes)]

However, this could even be marked as duplicate of LP 1874381 ...

[veribaka (veribaka)]

I thought so too initially, but the proposed solution doesn't work in my case, and also I get a different error message booting up.

[bene (analog2342)]

exactly the same happened here. Help would be much appreciated!

thanks

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[veribaka (veribaka)]

Small update. After several tests, I found that if I duplicate the encrypted block device configuration line in /etc/crypttab, the upgrade is successful.

I.e., in my case:

nvme0n1p3_crypt UUID=c9ffc13a-cad1-49a2-83eb-fb9002b3ebff none luks

changes to

nvme0n1p3_crypt UUID=c9ffc13a-cad1-49a2-83eb-fb9002b3ebff none luks
nvme0n1p3_crypt UUID=c9ffc13a-cad1-49a2-83eb-fb9002b3ebff none luks

I haven't tested if simply updating the timestamp on the file also does the trick.

Note: if doing this after the upgrade, it requires to re-generate initramfs.

[veribaka (veribaka)]

After more testing, it seems the file is not read because the line in /etc/crypttab doesn't have end of line.

This returns empty:
hexdump /etc/crypttab | grep 0a

This adds end of line, fixing the problem:
sudo sed -i -e '$a\' /etc/crypttab

[Nate (itsnotallitscrackeduptobe)]

I've encountered a similar issue for a root volume group w/ encryption. How can I try the same workaround as @veribaka starting from a busybox prompt? In busybox /etc/crypttab does not exist.

[Michel Bretschneider (michelbretschneider)]

So I upgraded from 18.04 LTS to 20.04 LTS too and found that the installed 5.4 kernel panics complaining something about volume group. I booted the 4.15 kernel wich put in a volume group warning too but at least runs. After adding 8.8.8.8 to the resolv.conf the machine is almost ready to work with (yup, I run my own dns/dhcp/dovecot/postfix which all need attention once that kernel/vg bug is no longer in the way).

Since my /boot partition is small, I initially thought the ramdisk image is missing/broken/corrupted (there were some warnings in the upgrade process which got stuck between cannot upgrade and cannot restore to initial state - which let me to reboot the machine, trying to fix that on my own). So I uninstalled the 4.4 kernel so only 4.15 and 5.4 are left. While grub was updating I noticed several

 WARNING: PV /dev/sda1 in VG nameOfVolumeGroup is using an old PV header, modify the VG to update.

So I ran sudo vgck --updatemetadata nameOfVolumeGroup to fix that, hoping it would suffice to make 5.4 boot. It didn't:

 Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

So I booted the 4.15 which complained

 Volume Group not found nameOfVolumeGroup
 Cannot process volume group nameOfVolumeGroup

but at least booted. Once the machine is booted the volume seems to be accessible, not sure why. Not sure why that didn't work for 5.4.

I checked /etc/crypttab but as I don't have crypted drives it contains only the comment header, hexdump -C shows it ends with 0a so no problem there.

Now I'm a bit lost:

- why is grub still giving me the option to boot the 4.4 kernel, although it's uninstalled?
- why does 5.4 fail to process the volume group and panic?
- why does 4.15 complain about the vg but still manages to start? (the only good thing so far)
- what should I do to make 5.4 (and updated kernels in the future) work on this system?
- why does dns does not fall back to 8.8.8.8 or something if the local dns got broken by the update (yeah, that's another issue - point me in the right direction or I'll search for that once the kernel/vg foo is fixed)

[Michel Bretschneider (michelbretschneider)]

Update: I installed 5.4.0-73 via update, grub config got updated but after reboot only

5.4.0-72 (the broken one)
4.15
4.4 (the uninstalled one)

appeared in the grub menu. After "reparing" the grub update (see: https://askubuntu.com/a/1207143), the machine boots into 5.4.0-73 without major issues. There are still a bunch of warnings but at least it starts.

[Michel Bretschneider (michelbretschneider)]

FYI: my system is quite old, I started with 8.04 server LTS and upgraded since so my problems are rarely reproducible on systems that were conceived later.