LVM device unavailable after 18.04 to 20.04 upgrade Timed out waiting for device /dev/mapper/s5lp8--v g-home

Bug #1874381 reported by John George
30
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
High
Skipper Bug Screeners
lvm2 (Ubuntu)
High
Canonical Foundations Team
Focal
High
Unassigned

Bug Description

After upgrading an LPAR configured with LVM from 18.04 to 20.04 /home is no longer mounted.
During boot the console shows Timed out waiting for device /dev/mapper/s5lp8--vg-home

Please see the attached dist-upgrade logs and console output for more detail.

Revision history for this message
John George (jog) wrote :
Revision history for this message
John George (jog) wrote :
tags: added: rls-ff-incoming
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Confirmed
Revision history for this message
Serhiy Zahoriya (xintx-ua) wrote :

See https://bugs.launchpad.net/bugs/1877473 for workarounds. While this report was first I was able to provide workarounds. Would you agree to mark this bug as a duplicate, please?

Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → High
Frank Heimes (fheimes)
tags: added: installer
Revision history for this message
Serhiy Zahoriya (xintx-ua) wrote :

Ok, copying my workarounds here and closing the other report:

Volume Group metadata had to be upgraded manually with

vgck --updatemetadata yourvgname

The VG name was somehow changed from lvm-name to just name and I had to edit /etc/crypttab and update-initramfs -u -k all

Also cryptsetup-initramfs package was deleted for some reason, had to re-install it in chroot. Maybe it happened because I have disabled installing recommended and suggested packages or maybe because none of *buntu-desktop packages are installed on my system. Not sure if I have to report it as a separate problem, please advise.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Confirmed
importance: Undecided → High
tags: removed: installer
Changed in ubuntu-release-upgrader (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu-release-upgrader (Ubuntu) → lvm2 (Ubuntu)
tags: removed: rls-ff-incoming
Changed in lvm2 (Ubuntu Focal):
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote :

John, are you able to confirm that a 'sudo vgck --updatemetadata s5lp8-vg' fixes this for you?

Also, before running that command, can you post the output of 'sudo vgdisplay' on this system?

Revision history for this message
Steve Langasek (vorlon) wrote :

You might also want to try 'sudo vgck -t s5lp8-vg' first before committing changes with --updatemetadata.

Changed in lvm2 (Ubuntu Focal):
milestone: none → ubuntu-20.04.1
tags: added: id-5ec6a06d9ddd891b690ef2e4
Revision history for this message
veribaka (veribaka) wrote :

I too tried to upgrade from 18.04 and ended up with a busybox, Ubuntu not finding my vg. Not sure how to enter those commands from a busybox Steve.

Re-imaging the device with 18.04 for the time being.

Revision history for this message
Serhiy Zahoriya (xintx-ua) wrote :

You need to either execute these commands after upgrade and _before_ rebooting or chroot from a LiveUSB boot.

Revision history for this message
veribaka (veribaka) wrote :

Reporting back:

"sudo vgck -t vg" reports that no changes would be made, it seems to be looking in /dev/sda rather than /dev/nvme0n1.

"sudo vgscan" first reports that /dev/sda open failed and then successfully detects the vg using metadata type lvm2.

Revision history for this message
Steve Langasek (vorlon) wrote :

awaiting feedback from the original reporter to confirm that these particular commands are applicable to his install.

Changed in lvm2 (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
veribaka (veribaka) wrote :

Is there a time frame to consider that the original reporter will not be returning to this discussion?

Should I open my own bug report with my logs?

Revision history for this message
Brian Murray (brian-murray) wrote : Re: [Bug 1874381] Re: LVM device unavailable after 18.04 to 20.04 upgrade Timed out waiting for device /dev/mapper/s5lp8--v g-home

On Fri, Jun 05, 2020 at 09:38:35AM -0000, veribaka wrote:
> Is there a time frame to consider that the original reporter will not be
> returning to this discussion?
>
> Should I open my own bug report with my logs?

Please do open your own bug report with the relevant log files and make
a comment in this bug report with the number of your new report.

Thanks!
--
Brian Murray

Revision history for this message
John George (jog) wrote :

I redeployed the system where this failures was original observed and recreated the issue.
Running 'sudo vgck --updatemetadata s5lp8-vg' did work around this bug, when run after the upgrade but prior to the reboot.

Here is the vgdisplay output requested by Steve:

ubuntu@s5lp8:~$ sudo vgdisplay
  --- Volume group ---
  VG Name s5lp8-vg
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 4
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 3
  Open LV 3
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size <64.00 GiB
  PE Size 4.00 MiB
  Total PE 16383
  Alloc PE / Size 16377 / 63.97 GiB
  Free PE / Size 6 / 24.00 MiB
  VG UUID 3B7wkv-AIvn-5A7N-pmWB-7y7h-ylkd-jUIfRd

Changed in lvm2 (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

Revision history for this message
veribaka (veribaka) wrote :
tags: added: fr-81
Changed in lvm2 (Ubuntu Focal):
milestone: ubuntu-20.04.1 → ubuntu-20.04.2
Changed in lvm2 (Ubuntu Focal):
milestone: ubuntu-20.04.2 → focal-updates
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

> So one solution to this would be to run "vgck --updatemetadata" on each vg in postinst... would there be downside to that?

how slow is it on very very large LVMs? Is it distructive? Maybe something to check with devops-y people, like I.S. / bootstack?

Revision history for this message
Frank Heimes (fheimes) wrote :

Sorry, I can't answer that for huge LVM installations.
But what I can say is that it is for s390x installations (multipath with LVMs on top) a kind of immediately responding.

One thought is to check/do this for example at the end of the dist-upgrade process, where a potential delay wouldn't be a very huge drawback (compared to a system that doesn't come up again after reboot).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers