system doesn't properly boot as expected if /usr is on its own LV

Bug #1854981 reported by Eric Desrochers on 2019-12-03
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Disco
Undecided
Unassigned
lvm2 (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Medium
Eric Desrochers
Disco
Medium
Eric Desrochers

Bug Description

[Impact]
In summary, the problem is a LVM 'auto-activation' issue. Basically the LVM activation doesn't occurs during the boot process (except for LVM root partition) and since /usr is an important component before the 'pivot'[0], the init inside the initramfs space tries to mount /usr. If /usr is a LVM volume then it will try to mount it while its backend device is not activated yet (due to the actual auto-activation issue), thus the mounting will fails and bring the user to the initramfs prompt for debugging.

Workaround:
At the initramfs prompt: lvm vgchange -ay and then 'ctrl-d' to resume the boot process.

[Test Case]

- Install Bionic or Disco
- Create a VG with /usr on its on LV
- Complete the installation
- At reboot the system will be stuck at mounting /usr file system

Extra notes:
Since this problem is ONLY related to LVM activation. If one create a non-LVM /usr separate partition, everything will work as expected.

Other separate LVM (e.g. /home, /var/, /srv) if on its own separate LVM volume will not be activate either, but since they don't need to be mounted in the initramfs space, it's not a problem, they can be activated later on (after the pivot) by systemd.

So the only combination possible is when /usr is on its separate LV only.

[Regression potential]

I don't foresee any regression, the fix will instruct udev rule to pvscan activate volume based on their major/minor number if LVM is scanned.

+RUN+="(LVM_EXEC)/lvm pvscan --cache --activate ay --major $major --minor $minor", ENV{LVM_SCANNED}="1"

The current situation doesn't auto-activate at all.

One downside, for users who will see this fixed, will still experience the situation in standard ISO, since I'm afraid Disco won't produces new ISO. (it is not recommended to do new disco installs anyway since it's going EOL shortly) One would need to rely on the mini.iso for fetching the latest lvm2 package in the archive.

IIRC Bionic will have a new point release somewhere in Feb 2020 but until then one would need to rely on the mini.iso for fetching the latest lvm2 package in the archive as well.

or use the workaround in the [Impact] section and then apt-get dist-upgrade in order to get the latest lvm2.

[Other information]

This problem only exhibit in Bionic and Disco.
Xenial and Eoan and late didn't exhibit the situation.

https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/
https://wiki.debian.org/UsrMerge

[Original Description]

Only the lv for root volume get activated, because of the grub parameter that specifies/enforce it "root=/dev/mapper/ubuntu-vg-ubuntu--lv"

http://man7.org/linux/man-pages/man7/bootparam.7.html
'root=...'
              This argument tells the kernel what device is to be used as
              the root filesystem while booting.

If one add a separate LV for /usr, the system will go straight to initramfs prompt failling to mount /usr.

At initramfs prompt, we notice that 'lv-usr' status is 'NOT available'.

Performing 'lvm vgchange -ay' at initramfs prompt workaround the problem and allow one to successfully boot.

Adding 'debug' parameter, we clearly we see /root being detected and mounted:
initramfs.debug:
=> + mount -r -t ext4 /dev/mapper/ubuntu--vg-ubuntu--lv /root
=> + mountroot_status=0
+ [ ]
+ log_end_msg
+ _log_msg done.\n
+ [ n = y ]
+ printf done.\n
done.
+ read_fstab_entry /usr
+ found=1
+ [ -f /root/etc/fstab ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ / = /usr ]
+ read MNT_FSNAME MNT_DIR MNT_TYPE MNT_OPTS MNT_FREQ MNT_PASS MNT_JUNK
+ [ /usr = /usr ]
+ [ -n ]
+ found=0
+ break 2
+ return 0
+ log_begin_msg Mounting /usr file system
+ _log_msg Begin: Mounting /usr file system ...

then the code read /etc/fstab and specifically search for /usr (most likely because of the /usr binary merged) and try to mount if if found.

initramfs-tools:init
271 maybe_break mount
272 log_begin_msg "Mounting root file system"
273 # Always load local and nfs (since these might be needed for /etc or
274 # /usr, irrespective of the boot script used to mount the rootfs).
275 . /scripts/local
276 . /scripts/nfs
277 . /scripts/${BOOT}
278 parse_numeric "${ROOT}"
279 maybe_break mountroot
280 mount_top
281 mount_premount
282 mountroot
283 log_end_msg
284
=> 285 if read_fstab_entry /usr; then
=> 286 log_begin_msg "Mounting /usr file system"
=> 287 mountfs /usr
=> 288 log_end_msg
=> 289 fi

In this case, /usr is present in /etc/fstab but the logical volume is not available, so it is mounting a filesystem without his backend device, thus goes straight to the initramfs prompt because /usr couldn't be mounted.

It clearly seems to be an 'auto-activation' issue at boot.

For other such as /home, /var, it's not a big deal cause they can be activated later on in the process and they are (I haven't check but I guess systemd or systemd unit/generator is taking care of it at some point), but /usr is a important piece to have mounted before the pivot since it contains most of the crucial binary, especially that nowadays /sbin, /bin, and /lib are pointing to /usr:

lrwxrwxrwx 1 root root 10 Aug 26 13:51 libx32 -> usr/libx32
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib64 -> usr/lib64
lrwxrwxrwx 1 root root 9 Aug 26 13:51 lib32 -> usr/lib32
lrwxrwxrwx 1 root root 7 Aug 26 13:51 lib -> usr/lib
lrwxrwxrwx 1 root root 7 Aug 26 13:51 bin -> usr/bin
lrwxrwxrwx 1 root root 8 Aug 26 13:51 sbin -> usr/sbin

NOTE:

* That doesn't affect /usr found in /etc/fstab on its separate partition when no LVM involve (e.g. /dev/vdb /usr ext4 ....). It only cause issue when /usr is in a LVM context.

* I was able to reproduce on Bionic and Disco so far.
Eoan doesn't seem to exhibit the situation so far in my testing.

* While certain release such as Bionic, Xenial doesn't come implicitly with the /usr merge approach. One can install package 'usrmerge' and convert their system /usr merged. I don't think removing the read_fstable_entry for /usr is an option here, as some user could potentially decide to convert their system with 'usrmerge' pkg.

Eric Desrochers (slashd) on 2019-12-03
tags: added: sts
Eric Desrochers (slashd) wrote :

initramfs.debug_DISCO

Eric Desrochers (slashd) on 2019-12-03
description: updated
Eric Desrochers (slashd) on 2019-12-03
Changed in lvm2 (Ubuntu):
importance: Undecided → High
Changed in initramfs-tools (Ubuntu):
importance: Undecided → High
description: updated
Eric Desrochers (slashd) on 2019-12-03
description: updated
Eric Desrochers (slashd) on 2019-12-03
description: updated
description: updated
Eric Desrochers (slashd) wrote :

The problem seems to come from "/udev/69-dm-lvm-metad.rules.in".

Still investigating, stay tuned.

Eric Desrochers (slashd) on 2019-12-05
Changed in initramfs-tools (Ubuntu Disco):
status: New → Won't Fix
Changed in initramfs-tools (Ubuntu Bionic):
status: New → Won't Fix
Changed in initramfs-tools (Ubuntu Xenial):
status: New → Won't Fix
Changed in initramfs-tools (Ubuntu):
status: New → Won't Fix
Changed in lvm2 (Ubuntu):
status: New → Fix Released
Changed in lvm2 (Ubuntu Disco):
status: New → Confirmed
Changed in lvm2 (Ubuntu Bionic):
status: New → Confirmed
importance: Undecided → Medium
Changed in lvm2 (Ubuntu Disco):
importance: Undecided → Medium
Changed in lvm2 (Ubuntu):
importance: High → Undecided
Eric Desrochers (slashd) on 2019-12-05
Changed in lvm2 (Ubuntu Xenial):
status: New → Won't Fix
Eric Desrochers (slashd) on 2019-12-05
description: updated
description: updated
description: updated
description: updated
Eric Desrochers (slashd) on 2019-12-05
Changed in lvm2 (Ubuntu Bionic):
status: Confirmed → In Progress
Changed in lvm2 (Ubuntu Disco):
status: Confirmed → In Progress
assignee: nobody → Eric Desrochers (slashd)
Changed in lvm2 (Ubuntu Bionic):
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd) wrote :

Uploaded in B/D. Waiting for SRU team verification team to approve and allow packages to start building in -proposed.

- Eric

Eric Desrochers (slashd) on 2019-12-05
description: updated
Eric Desrochers (slashd) on 2019-12-05
Changed in lvm2 (Ubuntu Xenial):
status: Won't Fix → Fix Released

Hello Eric, or anyone else affected,

Accepted lvm2 into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lvm2/2.02.176-4.1ubuntu4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in lvm2 (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in lvm2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Łukasz Zemczak (sil2100) wrote :

Hello Eric, or anyone else affected,

Accepted lvm2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lvm2/2.02.176-4.1ubuntu3.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Eric Desrochers (slashd) on 2019-12-05
description: updated
description: updated

All autopkgtests for the newly accepted lvm2 (2.02.176-4.1ubuntu4.2) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

writeboost/unknown (armhf)
udisks2/unknown (armhf)
cinder/2:14.0.1-0ubuntu1 (i386, ppc64el, s390x, amd64, arm64, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#lvm2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted lvm2 (2.02.176-4.1ubuntu3.18.04.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/2.16.0~rc2-1build1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#lvm2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Eric Desrochers (slashd) wrote :

[VERIFICATION BIONIC]

I have installed Bionic, layout /usr on it's on LV, and at reboot same problem as describe in the impact section. Server stuck in trying to mount /usr due to lack of LVM activation.

At the initramfs prompt, I ran 'lvm vgchange -ay' to boot properly. I installed the bionic-proposed package of lvm2[0], reboot the system and everything boot fine. The udev rule fix does the work correctly.

[0] dpkg -l
ii liblvm2app2.2:amd64 2.02.176-4.1ubuntu3.18.04.2 amd64 LVM2 application library
ii liblvm2cmd2.02:amd64 2.02.176-4.1ubuntu3.18.04.2 amd64 LVM2 command library
ii lvm2 2.02.176-4.1ubuntu3.18.04.2 amd64 Linux Logical Volume Manage

tags: added: verification-done-bionic
removed: verification-needed-bionic
Eric Desrochers (slashd) wrote :

I have restart the autopkgtest and they all passes BUT the disco cinder ones.

Looking at the failure, it's not related to this current SRU, and it seems to be a recurrent failing pattern:

autopkgtest [04:07:34]: test cinder-daemons: [-----------------------
2019-12-06 04:07:37.800 12844 WARNING oslo_db.sqlalchemy.engines [-] URL mysql://cinder:***@localhost/cinder does not contain a '+drivername' portion, and will make use of a default driver. A full dbname+drivername:// protocol is recommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility
2019-12-06 04:07:37.803 12844 CRITICAL cinder [-] Unhandled error: ModuleNotFoundError: No module named 'MySQLdb'
2019-12-06 04:07:37.803 12844 ERROR cinder Traceback (most recent call last):
......
2019-12-06 04:07:37.803 12844 ERROR cinder File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 102, in dbapi
2019-12-06 04:07:37.803 12844 ERROR cinder return __import__('MySQLdb')
2019-12-06 04:07:37.803 12844 ERROR cinder ModuleNotFoundError: No module named 'MySQLdb'
2019-12-06 04:07:37.803 12844 ERROR cinder 
autopkgtest [04:07:38]: test cinder-daemons: -----------------------]
autopkgtest [04:07:39]: test cinder-daemons: - - - - - - - - - - results - - - - - - - - - -
cinder-daemons FAIL non-zero exit status 1

Please ignore the following disco failures:

Regression in autopkgtest for cinder (ppc64el): test log
Regression in autopkgtest for cinder (s390x): test log
Regression in autopkgtest for cinder (amd64): test log
Regression in autopkgtest for cinder (arm64): test log
Regression in autopkgtest for cinder (i386): test log
Regression in autopkgtest for cinder (armhf): test log

Its failing because it can't find the MySQLdb module which is unrelated to the current LVM2 SRU.

Eric Desrochers (slashd) wrote :

[VERIFICATION DISCO]

I have installed Disco, layout /usr on it's on LV, and at reboot same problem as describe in the impact section. Server stuck in trying to mount /usr due to lack of LVM activation.

At the initramfs prompt, I ran 'lvm vgchange -ay' to boot properly. I installed the disco-proposed package of lvm2[0], reboot the system and everything boot fine. The udev rule fix does the work correctly.

[0] dpkg -l
ii liblvm2app2.2:amd64 2.02.176-4.1ubuntu4.2 amd64 LVM2 application library
ii liblvm2cmd2.02:amd64 2.02.176-4.1ubuntu4.2 amd64 LVM2 command library
ii lvm2 2.02.176-4.1ubuntu4.2 amd64 Linux Logical Volume Manager
ubuntu

Additionally, as stated in comment #13, the autopkgtest failure can be ignored as they are unrelated to this SRU.

tags: added: verification-done-disco
removed: verification-needed-disco

The verification of the Stable Release Update for lvm2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lvm2 - 2.02.176-4.1ubuntu3.18.04.2

---------------
lvm2 (2.02.176-4.1ubuntu3.18.04.2) bionic; urgency=medium

  * d/p/fix-auto-activation-at-boot.patch: (LP: #1854981)
    Allow LV auto-activation (e.g. /usr on it's separate LV)

 -- Eric Desrochers <email address hidden> Thu, 05 Dec 2019 15:15:56 +0000

Changed in lvm2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lvm2 - 2.02.176-4.1ubuntu4.2

---------------
lvm2 (2.02.176-4.1ubuntu4.2) disco; urgency=medium

  * d/p/fix-auto-activation-at-boot.patch: (LP: #1854981)
    Allow LV auto-activation (e.g. /usr on it's separate LV)

 -- Eric Desrochers <email address hidden> Thu, 05 Dec 2019 13:13:18 +0000

Changed in lvm2 (Ubuntu Disco):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers