LVM device unavailable after 18.04 to 20.04 upgrade Timed out waiting for device /dev/mapper/s5lp8--v g-home

[John George]

After upgrading an LPAR configured with LVM from 18.04 to 20.04 /home is no longer mounted.
During boot the console shows Timed out waiting for device /dev/mapper/s5lp8--vg-home

Please see the attached dist-upgrade logs and console output for more detail.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Serhiy (xintx-ua)]

See https://bugs.launchpad.net/bugs/1877473 for workarounds. While this report was first I was able to provide workarounds. Would you agree to mark this bug as a duplicate, please?

[Serhiy (xintx-ua)]

Ok, copying my workarounds here and closing the other report:

Volume Group metadata had to be upgraded manually with

vgck --updatemetadata yourvgname

The VG name was somehow changed from lvm-name to just name and I had to edit /etc/crypttab and update-initramfs -u -k all

Also cryptsetup-initramfs package was deleted for some reason, had to re-install it in chroot. Maybe it happened because I have disabled installing recommended and suggested packages or maybe because none of *buntu-desktop packages are installed on my system. Not sure if I have to report it as a separate problem, please advise.

[Steve Langasek (vorlon)]

John, are you able to confirm that a 'sudo vgck --updatemetadata s5lp8-vg' fixes this for you?

Also, before running that command, can you post the output of 'sudo vgdisplay' on this system?

[Steve Langasek (vorlon)]

You might also want to try 'sudo vgck -t s5lp8-vg' first before committing changes with --updatemetadata.

[veribaka (veribaka)]

I too tried to upgrade from 18.04 and ended up with a busybox, Ubuntu not finding my vg. Not sure how to enter those commands from a busybox Steve.

Re-imaging the device with 18.04 for the time being.

[Serhiy (xintx-ua)]

You need to either execute these commands after upgrade and _before_ rebooting or chroot from a LiveUSB boot.

[veribaka (veribaka)]

Reporting back:

"sudo vgck -t vg" reports that no changes would be made, it seems to be looking in /dev/sda rather than /dev/nvme0n1.

"sudo vgscan" first reports that /dev/sda open failed and then successfully detects the vg using metadata type lvm2.

[Steve Langasek (vorlon)]

awaiting feedback from the original reporter to confirm that these particular commands are applicable to his install.

[veribaka (veribaka)]

Is there a time frame to consider that the original reporter will not be returning to this discussion?

Should I open my own bug report with my logs?

[Brian Murray (brian-murray)]

On Fri, Jun 05, 2020 at 09:38:35AM -0000, veribaka wrote:
> Is there a time frame to consider that the original reporter will not be
> returning to this discussion?
>
> Should I open my own bug report with my logs?

Please do open your own bug report with the relevant log files and make
a comment in this bug report with the number of your new report.

Thanks!
--
Brian Murray

[John George (jog)]

I redeployed the system where this failures was original observed and recreated the issue.
Running 'sudo vgck --updatemetadata s5lp8-vg' did work around this bug, when run after the upgrade but prior to the reboot.

Here is the vgdisplay output requested by Steve:

ubuntu@s5lp8:~$ sudo vgdisplay
  --- Volume group ---
  VG Name s5lp8-vg
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 3
  Open LV 3
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size <64.00 GiB
  PE Size 4.00 MiB
  Total PE 16383
  Alloc PE / Size 16377 / 63.97 GiB
  Free PE / Size 6 / 24.00 MiB
  VG UUID 3B7wkv-AIvn-5A7N-pmWB-7y7h-ylkd-jUIfRd

[Michael Hudson-Doyle (mwhudson)]

So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

[veribaka (veribaka)]

Hello, please see https://bugs.launchpad.net/ubuntu-z-systems/+bug/1887964

[Dimitri John Ledkov (xnox)]

> So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

how slow is it on very very large LVMs? Is it distructive? Maybe something to check with devops-y people, like I.S. / bootstack?

[Frank Heimes (fheimes)]

Sorry, I can't answer that for huge LVM installations.
But what I can say is that it is for s390x installations (multipath with LVMs on top) a kind of immediately responding.

One thought is to check/do this for example at the end of the dist-upgrade process, where a potential delay wouldn't be a very huge drawback (compared to a system that doesn't come up again after reboot).