loading lvm2 module crashes: udisks_module_lvm2_new: assertion 'UDISKS_IS_DAEMON (daemon)' failed

[Martin Pitt]

[Impact]

Trying to load the lvm2 module crashes udisks immediately:

   busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/Manager org.freedesktop.UDisks2.Manager EnableModule sb lvm2 true

udisksd[5709]: cannot register existing type 'UDisksDaemon'
udisksd[5709]: g_once_init_leave: assertion 'result != 0' failed
udisksd[5709]: udisks_module_lvm2_new: assertion 'UDISKS_IS_DAEMON (daemon)' failed
systemd[1]: Started [e-mail] - Process Core Dump (PID 5767/UID 0).

                                              Module libudev.so.1 from deb systemd-253.5-1ubuntu6.amd64
                                               Module libsystemd.so.0 from deb systemd-253.5-1ubuntu6.amd64
                                               Stack trace of thread 5709:
                                               #0 0x000056079be701ad udisks_module_manager_load_modules (udisksd + 0x601ad)
                                               #1 0x000056079be5fdf5 n/a (udisksd + 0x4fdf5)
                                               #2 0x00007f7d5f122a11 n/a (libglib-2.0.so.0 + 0x5aa11)
                                               #3 0x00007f7d5f17e46f n/a (libglib-2.0.so.0 + 0xb646f)
                                               #4 0x00007f7d5f12346f g_main_loop_run (libglib-2.0.so.0 + 0x5b46f)
                                               #5 0x000056079be323d2 main (udisksd + 0x223d2)
                                               #6 0x00007f7d5ee280d0 __libc_start_call_main (libc.so.6 + 0x280d0)
                                               #7 0x00007f7d5ee28189 __libc_start_main_impl (libc.so.6 + 0x28189)
                                               #8 0x000056079be324f5 _start (udisksd + 0x224f5)

This breaks at least cockpit, and possibly other consumers of udisks (GNOME Disks perhaps?). This is a regression in mantic, earlier releases were fine.

[Test Plan]

The above reproducer confirms the check. The busctl command should succeed, and afterwards `systemctl status udisks2` should still be "active (running)" instead of "failed".

autopkgtests should take care of regression testing, and they were happy with the noble update. I'm also happy to run cockpit's integration tests against the updated version; it exercises udisks2 thoroughly (and we find lots of bugs with it). At the moment we don't test 23.10 mantic at all, because it is so broken (https://github.com/cockpit-project/bots/pull/5432).

[Where problems could occur]

Changing the linker options is quite invasive. The autopkgtests cover the runtime functionality though, and as modules are currently 100% broken they can't possibly get any worse.

[Other info]

Fix committed to Debian: https://salsa.debian.org/utopia-team/udisks2/-/commit/e22881abdd90bd4878add165ccc46008a17163f8

Noble debdiff: http://launchpadlibrarian.net/701038246/udisks2_2.10.1-1ubuntu1_2.10.1-1ubuntu2.diff.gz

ProblemType: Crash
Architecture: amd64
Date: Wed Oct 25 12:13:14 2023
DistroRelease: Ubuntu 23.10
ExecutablePath: /usr/libexec/udisks2/udisksd
Package: udisks2 2.10.1-1ubuntu1

[Martin Pitt (pitti)]

Confirmed on current noble, same udisks2 version.

[Martin Pitt (pitti)]

It's a bit easier to investigate with

    /usr/libexec/udisks2/udisksd --debug --force-load-modules

There I've1 also seen it fail on the btrfs module, so it's not specific to the LVM one.

It also happens with rebuilding the udisks source package (so not just changed ABI)

I tried building upstream git, but it fails due to some missing API, like

udiskslinuxmodulelvm2.c:438:29: error: implicit declaration of function 'udisks_logical_volume_get_uuid' [-Werror=implicit-function-declaration]

presumably libblockdev or some other build dep which moved along?

Anyway, this doesn't happen on Debian, and it's certainly not related to the current Ubuntu delta. So something in the stack must be different. glib2.0 would be the obvious candidate, but both Debian sid and Ubuntu noble have 2.78.1-4.

For investigating it is helpful to load the modules from the source tree:

--- src/udisksmodulemanager.c.orig 2023-12-04 10:20:54.571542454 +0000
+++ src/udisksmodulemanager.c 2023-12-04 10:21:01.047569622 +0000
@@ -637,7 +637,7 @@
 udisks_module_manager_new (UDisksDaemon *daemon)
 {
   return UDISKS_MODULE_MANAGER (g_object_new (UDISKS_TYPE_MODULE_MANAGER,
- "daemon", daemon, NULL));
+ "daemon", daemon, "uninstalled", TRUE, NULL));
 }

 /**

and then

DEB_BUILD_OPTIONS=nostrip CFLAGS="-g -O0" dpkg-buildpackage -us -uc -b -j4 -Pnodoc

for iterating:

make && sudo G_DEBUG=fatal-criticals LD_LIBRARY_PATH=udisks/.libs/ gdb --args src/.libs/udisksd --debug --force-load-modules

[Martin Pitt (pitti)]

Comparing with Debian testing: Running

# /usr/libexec/udisks2/udisksd --debug --force-load-modules
udisks-Message: 10:34:44.696: udisks daemon version 2.10.1 starting
udisks-Message: 10:34:44.724: Acquired the name org.freedesktop.UDisks2 on the system message bus
udisks-Message: 10:34:44.729: Loading module lvm2 ...

there works, and doesn't show any of the three assertions that fail on noble:

(udisksd:57782): GLib-GObject-CRITICAL **: 10:33:06.459: cannot register existing type 'UDisksDaemon'

(udisksd:57782): GLib-CRITICAL **: 10:33:06.460: g_once_init_leave: assertion 'result != 0' failed

(udisksd:57782): libudisks2-lvm2-CRITICAL **: 10:33:06.460: udisks_module_lvm2_new: assertion 'UDISKS_IS_DAEMON (daemon)' failed

glib2.0 and libblockdev have identical versions.

Installing debian-testing's glib debs:

  apt install --reinstall libglib2.0-{0,bin,data,dev,dev-bin,doc}

and/or the libblockdev debs:

  dpkg -l | grep 3.0.4-1 | awk '{print $2}' | xargs apt install -y --reinstall

makes no difference. I even tried to install Debian testing's binutils.

Conversely, mantic's/noble's udisks debs don't install on debian testing because

dpkg: dependency problems prevent configuration of udisks2:
 udisks2 depends on libc6 (>= 2.38); however:
  Version of libc6:amd64 on system is 2.37-12.

a-ha!

But upgrading libc6 to debian experimental (2.38-4) on debian testing image still works fine.

[Martin Pitt (pitti)]

Interesting! when I check out the source package, and build with the above patch and like this:

  ./configure CFLAGS="-g -O0" --disable-gtk-docs --enable-lvm2
  make -j4
  sudo src/udisksd -rd --force-load-modules

then it loads the module just fine:

(udisksd:17050): udisks-WARNING **: 14:52:41.121: Can't load configuration file /usr/local/etc/udisks2/udisks2.conf
udisks-Message: 14:52:41.122: Loading module lvm2 ...
udisks-Message: 14:52:41.131: Acquired the name org.freedesktop.UDisks2 on the system message bus

Curiously, when I run the exact same ./configure invocation that the Debian packaging does, and run it out of the build tree, it works.

It does seem to be *something* about the built module, though -- with the patch above (running modules from build tree) it works, but without the patch it loads them from /usr and fails.

Back trace:

#0 0x00007ffff79a8a5d in g_logv () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#1 0x00007ffff79a8cf3 in g_log () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007ffff7ac2ead in ??? () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#3 0x00007ffff7aca18f in g_type_register_static () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#4 0x00007ffff7aca3e8 in g_type_register_static_simple () at /lib/x86_64-linux-gnu/libgobject-2.0.so.0
#5 0x00007ffff46da71d in udisks_daemon_get_type_once () at udisksdaemon.c:115
        g_define_type_id = 0x1
#6 0x00007ffff46da69c in udisks_daemon_get_type () at udisksdaemon.c:115
        g_define_type_id = Python Exception <class 'ValueError'>: Variable 'static_fundamental_type_nodes' not found.

        static_g_define_type_id = 0
#7 0x00007ffff46cb784 in udisks_module_lvm2_new (daemon=0x555555615340, cancellable=0x0, error=0x7fffffffe210)
    at udiskslinuxmodulelvm2.c:133
        __inst = 0x555555615340
        __t = Python Exception <class 'gdb.error'>: No type named TypeNode.

        __r = 32767
        initable = 0x5555556131b0
        __func__ = "udisks_module_lvm2_new"
#8 0x00005555555d21a3 in load_single_module_unlocked
    (manager=0x555555624950, sopath=0x555555637090 "/usr/lib/x86_64-linux-gnu/udisks2/modules/libudisks2_lvm2.so", do_notify=0x7fffffffe208, error=0x7fffffffe210) at udisksmodulemanager.c:344
        state = 0x5555555eb6b5
        handle = 0x5555556a2450
        module_id = 0x555555692ea0 "lvm2"
        module_new_func_name = 0x555555613ae0 "\2602v\364\377\177"
        module_id_func = 0x7ffff46cb73a <udisks_module_id>
        module_new_func = 0x7ffff46cb75f <udisks_module_lvm2_new>
        module = 0x5555556a3693
        __func__ = "load_single_module_unlocked"
#9 0x00005555555d2505 in udisks_module_manager_load_modules (manager=0x555555624950) at udisksmodulemanager.c:448
        modules_to_load = 0x5555556a2150 = {0x555555637090}
        modules_to_load_tmp = 0x5555556a2150 = {0x555555637090}
        error = 0x0
        do_notify = 0
        __func__ = "udisks_module_manager_load_modules"
#10 0x0000555555578002 in load_modules_in_idle_cb (user_data=0x555555615340) at udisksdaemon.c:277
        daemon = 0x555555615340
#11 0x00007ffff79a0a61 in ??? () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#12 0x00007ffff79fc4ff in ??? () at /lib/x86_6...

[Martin Pitt (pitti)]

It's not related to the configure arguments, but to the linker flags. The Ubuntu package build sets

  LDFLAGS="-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro -Wl,-z,now"

while Debian uses

  LDFLAGS="-Wl,-z,relro -Wl,-z,now"

lto seems to work, but dropping the "-Wl,-Bsymbolic-functions" makes it work. But that was already enabled ages ago, at least since intrepid in 2008.

Reproduces with

  ./configure LDFLAGS="-Wl,-Bsymbolic-functions" --disable-gtk-docs --enable-lvm2 && make -j4
  sudo src/udisksd --debug --force-load-modules --replace

This is actually similar to bug #1286046 (fixed in https://launchpad.net/ubuntu/+source/parole/0.6.1-0ubuntu2) or https://github.com/fwupd/fwupd/issues/1077 -- in both cases, -Bsymbolic-functions got dropped to fix it.

Adding

  export DEB_LDFLAGS_MAINT_STRIP=-Wl,-Bsymbolic-functions

fixes it. I commited that to Debian [1], and uploaded.

[1] https://salsa.debian.org/utopia-team/udisks2/-/commit/e22881abdd90bd4878add165ccc46008a17163f8

[Launchpad Janitor (janitor)]

This bug was fixed in the package udisks2 - 2.10.1-1ubuntu2

---------------
udisks2 (2.10.1-1ubuntu2) noble; urgency=medium

  * Don't build with -Wl,-Bsymbolic-functions. It breaks module loading, as
    it confuses GObject's type loading cache: "cannot register existing type
    'UDisksDaemon'". (Patch also applied to Debian packaging git, can be
    synced next time). (LP: #2040488)

 -- Martin Pitt <email address hidden> Tue, 05 Dec 2023 11:25:08 +0000

[Martin Pitt (pitti)]

Exact same fix uploaded to -proposed SRU review queue.

[Timo Aaltonen (tjaalton)]

Hello Martin, or anyone else affected,

Accepted udisks2 into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/udisks2/2.10.1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Martin Pitt (pitti)]

I installed the build from mantic-proposed and confirm that the simple test case in the description now works. Moreover, in https://github.com/cockpit-project/bots/pull/5432 I ran cockpit's extensive storage tests (which exercise udisks a lot): Compared to the complete disaster with the released version 2.10.1-1ubuntu1 this now looks mostly green (except for a known and unrelated LVM2 regression) [1]

For some reason the autopkgtests didn't run yet on this update to do reverse dependency regression testing: https://ubuntu-archive-team.ubuntu.com/proposed-migration/mantic/update_excuses.html#udisks2 Apparently these need to be approved manually first?

[1] https://cockpit-logs.us-east-1.linodeobjects.com/pull-5432-20231210-095411-5680c888-ubuntu-stable-storage-cockpit-project-cockpit/log.html

[Brian Murray (brian-murray)]

The autopkgtests for udisks2 did run for mantic they are just not appearing in the excuses report for unknown reasons.

[Launchpad Janitor (janitor)]

This bug was fixed in the package udisks2 - 2.10.1-1ubuntu1.1

---------------
udisks2 (2.10.1-1ubuntu1.1) mantic-proposed; urgency=medium

  * Don't build with -Wl,-Bsymbolic-functions. It breaks module loading, as
    it confuses GObject's type loading cache: "cannot register existing type
    'UDisksDaemon'". (Patch also applied to Debian packaging git, can be
    synced next time). (LP: #2040488)

 -- Martin Pitt <email address hidden> Tue, 05 Dec 2023 11:25:08 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for udisks2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.