Multipath device's targets are not added to domain namespace/cgroup

[Bhavin Gandhi]

This is a bug in libvirt which has been fixed in upstream: https://gitlab.com/libvirt/libvirt/-/issues/790

> When using qemu-pr-helper with multipath devices, the target devices are not added to the domain's namespace as well as cgroups. As a result, the sg_persist calls from the guest VM fail.

And the fix will be available in the next release (v11.6.0), but needs to be backported to 22.04 and 24.04 as those are on 8.0.0 and 10.0.0 respectively.

I'm working on patch for 22.04 followed by 24.04

---

[ Impact ]

When using qemu-pr-helper with multipath devices, the target devices are not added to the domain's namespace as well as cgroups. As a result, the sg_persist calls from the guest VM fail. These are mostly used by clustered applications like Windows Cluster.

[ Test Plan ]

These steps help to reproduce the issue with the scsi_debug kernel module. We create a fake multipath device with 4 target disks, without the fix we can only see the device mapper device in qemu-pr-helper's mount namespace.

1. Get the scsi_debug kernel module and verify it is available.

   sudo apt install linux-modules-extra-$(uname -r)

   $ find /lib/modules/ -name "*scsi*" | grep debug
   /lib/modules/6.8.0-79-generic/kernel/drivers/scsi/scsi_debug.ko.zst

2. Ensure multipath-tools in installed on the system and multipathd service is running.

   sudo apt install multipath-tools
   sudo systemctl status multipathd.service

3. Create a multipath device with 4 target disks using scsi_debug

   sudo modprobe scsi_debug dev_size_mb=100 num_tgts=1 vpd_use_hostno=0 add_host=4 delay=20 max_luns=2 no_lun_0=1

   $ sudo multipath -ll
   mpatha (1Linux_scsi_debug_2001) dm-0 Linux,scsi_debug
   size=100M features='0' hwhandler='1 alua' wp=rw
   |-+- policy='service-time 0' prio=1 status=enabled
   | `- 2:0:0:1 sda 8:0 active ready running
   |-+- policy='service-time 0' prio=1 status=enabled
   | `- 5:0:0:1 sdd 8:48 active ready running
   |-+- policy='service-time 0' prio=50 status=active
   | `- 3:0:0:1 sdb 8:16 active ready running
   `-+- policy='service-time 0' prio=50 status=enabled
     `- 4:0:0:1 sdc 8:32 active ready running

   Notice that the name is 1Linux_scsi_debug_2001 and it has 4 disks, sda to sdd.

   We will use the following path of the multipath device mapper disk when creating the VM, this points to dm-0.

   /dev/disk/by-id/dm-uuid-mpath-1Linux_scsi_debug_2001

   $ ls -l /dev/disk/by-id/dm-uuid-mpath-1Linux_scsi_debug_2001
   … /dev/disk/by-id/dm-uuid-mpath-1Linux_scsi_debug_2001 -> ../../dm-0

4. Create a new VM with reservations (starts qemu-pr-helper along with the qemu process for the VM). Use domain.xml from attachment: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2117467/+attachment/5911261/+files/domain.xml

   Install libvirt first:

   sudo apt install libvirt-daemon libvirt-daemon-system libvirt-clients
   sudo systemctl start libvirtd

   Create the VM:

   sudo virsh define --file domain.xml --validate
   sudo virsh start libvirt-lp2117467

   Verify it is in running state.

   sudo virsh list --all

5. Find the mount namespace of the VM, enter the namespace and verify the disks are added there.

   Find the PID of qemu-pr-helper or qemu-system-x86

   $ pgrep qemu-pr-helper
   734148

   Enter the mount namespace of this process to verify all the target disks are present. In this example, we don't see the sda to sdd as libvirt is not patched.

   $ sudo nsenter --target 734148 --mount bash
   root@bhavin-build:/# ls /dev/
   disk dm-0 full hugepages mapper mqueue null ptmx pts random shm urandom zero

   When the patched version is installed and VM is re-created, we should be able to see the target devices (sda to sdd) as well.

   root@bhavin-build:/# ls /dev/
   block disk dm-0 full hugepages mapper mqueue null ptmx pts random sda sdb sdc sdd shm urandom zero
   # notice the sda, sdb, sdc, and sdd got added after installing update.

[ Where problems could occur ]

The bugfix is quite isolated to one particular area, the code handling device mapper references, things like /dev/disk/by-id/dm-* and what is can translate them to. Therefore the kind of potential regressions to look out for is exactly that, a handling of such targets in the guest configuration and how libvirt then uses them. Think of "device not found" or "can't access" for such devices due to
using the now correct names (in case there is a scenario where the old worked better).

[Bhavin Gandhi (bhavin192)]

Attaching patch for Jammy. I have used this change in my environment and it fixes the bug.

[Bhavin Gandhi (bhavin192)]

Patch for Noble.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "1-8.0.0-1ubuntu7.13.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Bhavin Gandhi (bhavin192)]

Is there anything I need to do for these patches to get applied or get any comments on these?
Or should I wait and these will be merged when it is the right time?

[Andreas Hasenack (ahasenack)]

Questing has 11.6.0 and I checked the code, the patch is applied. Marking that fix released. Confirmed noble and jammy lack the patch.

[Andreas Hasenack (ahasenack)]

Plucky also affected, of course.

[Lukas Märdian (slyon)]

Thanks for preparing the backports, Bhavin!
I can push this forward as part of the Server team's virtualization efforts.

Can you please help with updating the bug description according to the SRU template (i.e. describing a proper reproducer/testcase), so that we can drive the stable release updates? Thanks!

https://canonical-sru-docs.readthedocs-hosted.com/en/latest/reference/bug-template/

[Lukas Märdian (slyon)]

I reviewed the Noble debdiff. Patch matches upstream and applies cleanly.

The only thing I changed is moving up the patch in debian/patches/series to the "Ubuntu Changes" section, as this is not really related to "Ubuntu Apparmor Changes".

Merge-proposal and PPA for testing are up here: https://code.launchpad.net/~slyon/ubuntu/+source/libvirt/+git/libvirt/+merge/491493

[Bhavin Gandhi (bhavin192)]

Thank you for taking a look, was not aware of the SRU template. I was mostly testing in a specific environment where I have FC backed volumes with multipath. I'm looking into a way to reproduce this without any special setup requirements, so that I can fill up the testcase/reproducer part.

> The only thing I changed is moving up the patch in debian/patches/series to the "Ubuntu Changes" section, as this is not really related to "Ubuntu Apparmor Changes".

Ah, makes sense.

[Lukas Märdian (slyon)]

A generic reproducer would be great!
But testing it in your specific environment would also be fine, as long as it is described in detail (using the SRU template) and you'd be available for testing the changes for the different stable release updates on Plucky, Noble & Jammy.

It's a hard requirement that the final packages from -proposed would need to be tested on the specific Ubuntu version, before a SRU can be verified and released.

[Lukas Märdian (slyon)]

Hey, coming back to this case. Please don't put too much energy into finding a generic reproducer, it would be great to have, but reproducing it in your custom environment is good enough.

As long as you can help us verify the updated packages.
Would you be able to reproduce/verify the fix for Ubuntu 22.04, 24.04 and/or 25.10, potentially inside a corresponding VM inside your pre-configured environment?

If you can support us with the SRU verification, we could go ahead and upload those changes.

[Bhavin Gandhi (bhavin192)]

Sorry for no response for a while, I was away for a conference.
I can verify the fix only for Ubuntu 22.04 as of now.

Looks like I have found a way to test this locally, so let me take a day or two and get back with a reproducer and other details the SRU template needs.

[Bhavin Gandhi (bhavin192)]

I have updated the bug description according to SRU template. I can help with testing the changes on Jammy (in my environment) and Noble using the reproducer I added.

[Christian Ehrhardt (paelzer)]

Thank you Bhavin,
I tested the repro steps on plucky to fill the gap to do so when processing the SRU.
I've done so with a plucky VM and a second level guest to ensure it works there which means it could be verified by almost anyone.

Initial setup worked:

$ sudo multipath -ll
mpatha (1Linux_scsi_debug_2001) dm-0 Linux,scsi_debug
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 6:0:0:1 sda 8:0 active ready running
| `- 9:0:0:1 sdd 8:48 active ready running
|-+- policy='service-time 0' prio=10 status=enabled
| `- 7:0:0:1 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 8:0:0:1 sdc 8:32 active ready running

Slightly different names

$ ll /dev/disk/by-id/scsi-1Linux_scsi_debug_2001
lrwxrwxrwx 1 root root 10 Sep 24 08:00 /dev/disk/by-id/scsi-1Linux_scsi_debug_2001 -> ../../dm-0

The disk config element for the guest I used is:
I needed to add reservations here to get qemu-pr-helper spawned, which was not in your example AFAICS.

    <disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/scsi-1Linux_scsi_debug_2001'>
        <reservations managed='yes'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

With that I can see both spawned

6 64055 2293 1 20 0 1427788 572428 - Sl ? 0:18 /usr/bin/qemu-system-x86_64 ...
4 0 2296 1 20 0 89856 6732 - Sl ? 0:00 /usr/bin/qemu-pr-helper -k ...

In the namespace I can see the device already:

$ sudo nsenter --target 2296 --mount bash
$ ll /dev/disk/by-id/scsi-1Linux_scsi_debug_2001
lrwxrwxrwx 1 root root 26 Sep 24 09:36 /dev/disk/by-id/scsi-1Linux_scsi_debug_2001 -> /dev/disk/by-id/../../dm-0
ll /dev/dm-0
brw-rw---- 1 libvirt-qemu kvm 252, 0 Sep 24 09:36 /dev/dm-0

So while the instructions have been awesome, they work for me with an un-fixed libvirt already.
Since it isn't failing at first, it can't be used to verify the case :-/
What am I missing?

[Bhavin Gandhi (bhavin192)]

Thanks for giving it a spin, Christian.

Right I somehow missed reservations key from the domain.xml which I uploaded.

> So while the instructions have been awesome, they work for me with an un-fixed libvirt already.
> Since it isn't failing at first, it can't be used to verify the case :-/
> What am I missing?

I see, being able to see dm-0 is expected. But we should also see the target/child devices i.e. sda, sdb, sdc and sdd.

Let me update the description to clarify that.

This is how it shows to me after installing the patched libvirt:

root@bhavin-build:/# ls /dev/
block disk dm-0 full hugepages mapper mqueue null ptmx pts random *sda sdb sdc sdd* shm urandom zero

[Christian Ehrhardt (paelzer)]

Thanks for the new XML and thanks for explaining what you expect to be added.

So to be extra sure you expect lsblk to list them (already the case) but also be available in the namespaced /dev

$ lsblk | grep '^sd'
sda 8:0 0 100M 0 disk
sdb 8:16 0 100M 0 disk
sdc 8:32 0 100M 0 disk
sdd 8:48 0 100M 0 disk

implies that those are used in the namespace but due to the issue they are not available as

$ ll /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

And the latter should be changed with the fix.
Thanks for the explanation, makes sense to me now

Trying that against the PPA builds...

Yes with the fix this becomes:

$ pgrep qemu-pr-helper
4865

$ sudo nsenter --target 4865 --mount bash
root@p-lp-2117467:/# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Sep 24 10:27 /dev/sda
brw-rw---- 1 root disk 8, 16 Sep 24 10:27 /dev/sdb
brw-rw---- 1 root disk 8, 32 Sep 24 10:27 /dev/sdc
brw-rw---- 1 root disk 8, 48 Sep 24 10:27 /dev/sdd

OK, testcase confirmed - awesome.
Therefore ready for the SRU AFAICS - thanks

[Bhavin Gandhi (bhavin192)]

Yes, that's correct.

Those are needed in the namespaced /dev but are missing due to the issue. And this results in failure during qemu-pr-helper's operations.

[Christian Ehrhardt (paelzer)]

Thanks, I've updated the SRU template in regard to where issues would occur.

I think it is thereby ready for an upload to -unapproved.

FYI We have another bug 2120278, targeting mostly the same releases, once that is fully ready (soon) we'll upload them together to do the related process steps just once.

[Christian Ehrhardt (paelzer)]

All uploaded to -unapproved and now waiting to be seen on SRU shift processing the queue.

[Julian Andres Klode (juliank)]

Hello Bhavin, or anyone else affected,

Accepted libvirt into plucky-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/11.0.0-2ubuntu6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-plucky to verification-done-plucky. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-plucky. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

Hello Bhavin, or anyone else affected,

Accepted libvirt into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/10.0.0-2ubuntu8.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

Hello Bhavin, or anyone else affected,

Accepted libvirt into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/8.0.0-1ubuntu7.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bhavin Gandhi (bhavin192)]

I was able to verify the changes on noble by following the reproducer from the bug description. Here are the steps I followed before update.

VM is already defined and in running state. We can see only dm-0 is visible in the /dev/ (the bug).

ubuntu@bhavin-noble:~$ sudo nsenter --target 8748 --mount bash
root@bhavin-noble:/# ls /dev/
disk dm-0 full hugepages mapper mqueue null ptmx pts random shm urandom zero

root@bhavin-noble:/# ls /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

Enabled the noble-proposed and upgraded libvirt packages to 10.0.0-2ubuntu8.9

ubuntu@bhavin-noble:~$ sudo vi /etc/apt/sources.list.d/ubuntu.sources
ubuntu@bhavin-noble:~$ sudo apt install libvirt-daemon libvirt-daemon-system libvirt-clients -t noble-proposed

ubuntu@bhavin-noble:~$ sudo virsh destroy libvirt-lp2117467
Domain 'libvirt-lp2117467' destroyed

ubuntu@bhavin-noble:~$ sudo virsh start libvirt-lp2117467
Domain 'libvirt-lp2117467' started

ubuntu@bhavin-noble:~$ pgrep qemu-pr-helper
12027

ubuntu@bhavin-noble:~$ sudo nsenter --target 12027 --mount bash
root@bhavin-noble:/# ls /dev/
block disk dm-0 full hugepages mapper mqueue null ptmx pts random sda sdb sdc sdd shm urandom zero

root@bhavin-noble:/# ls /dev/sd*
/dev/sda /dev/sdb /dev/sdc /dev/sdd

Both the dm-0 and sda to sdd have been added correctly. Fixes the bug.

[Bhavin Gandhi (bhavin192)]

Verified on jammy as well. Performed exact same steps as noble, sharing history as I lost command logs (tmux crashed).

ubuntu@jammy:~$ history
   14 sudo virsh define --file domain.xml --validate
   15 sudo virsh start libvirt-lp2117467
   17 pgrep qemu-pr-helper
   18 sudo nsenter --target 15728 --mount bash

Only dm-0 was visible at /dev. Enabled jammy-proposed and upgraded libvirt packages to 8.0.0-1ubuntu7.13

   24 sudo apt install libvirt-daemon libvirt-daemon-system libvirt-clients -t jammy-proposed
   26 sudo virsh destroy libvirt-lp2117467
   27 sudo virsh start libvirt-lp2117467
   28 pgrep qemu-pr-helper
   29 sudo nsenter --target 17416 --mount bash

ubuntu@jammy:~$ sudo nsenter --target 17416 --mount bash
root@jammy:/# ls /dev/
block dm-0 hugepages mqueue ptmx random sdb sdd urandom
disk full mapper null pts sda sdc shm zero

root@jammy:/# ls /dev/sd*
/dev/sda /dev/sdb /dev/sdc /dev/sdd

Both dm-0 and sda to sdd can be seen in /dev/. Fix verified.

[Bhavin Gandhi (bhavin192)]

I was able to verify on plucky. Same steps as noble.

ubuntu@plucky:~$ sudo nsenter --target 11699 --mount bash
root@plucky:/# ls /dev/
disk dm-0 full hugepages mapper mqueue null ptmx pts random shm urandom userfaultfd zero

root@plucky:/# ls /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

Enabled the plucky-proposed and upgraded libvirt packages to 11.0.0-2ubuntu6.3

ubuntu@plucky:~$ sudo apt install libvirt-daemon libvirt-daemon-system libvirt-clients -t plucky-proposed

ubuntu@plucky:~$ sudo virsh destroy libvirt-lp2117467
Domain 'libvirt-lp2117467' destroyed

ubuntu@plucky:~$ sudo virsh start libvirt-lp2117467
Domain 'libvirt-lp2117467' started

ubuntu@plucky:~$ pgrep qemu-pr-helper
13492

ubuntu@plucky:~$ sudo nsenter --target 13492 --mount bash
root@plucky:/# ls /dev/
block disk dm-0 full hugepages mapper mqueue null ptmx pts random sda sdb sdc sdd shm urandom userfaultfd zero

root@plucky:/# ls /dev/sd*
/dev/sda /dev/sdb /dev/sdc /dev/sdd

Both dm-0 and sda to sdd can be seen in /dev/. Fix verified.

[Christian Ehrhardt (paelzer)]

Thank you so much Bhavin - great display of collaboration on the process!

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 11.0.0-2ubuntu6.3

---------------
libvirt (11.0.0-2ubuntu6.3) plucky; urgency=medium

  [ Lukas Märdian ]
  * d/p/u/lp-2117467-virdevmapper-device-name-for-targets.patch:
    virdevmapper: Always use device name for finding targets. This ensures
    that all the target devices of a multipath device are added to the
    namespace/cgroup of the guest domain.
    Closes LP: #2117467.

  [ Hector Cao ]
  * d/p/u-aa/lp2079869-* : virt-aa-helper: Avoid duplicate when append rule
    (LP: #2120278)

 -- Hector Cao <email address hidden> Wed, 17 Sep 2025 01:13:17 +0200

[Julian Andres Klode (juliank)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 10.0.0-2ubuntu8.9

---------------
libvirt (10.0.0-2ubuntu8.9) noble; urgency=medium

  [ Bhavin Gandhi ]
  * d/p/u/lp-2117467-virdevmapper-device-name-for-targets.patch:
    virdevmapper: Always use device name for finding targets. This ensures
    that all the target devices of a multipath device are added to the
    namespace/cgroup of the guest domain.
    Closes LP: #2117467.

  [ Hector Cao ]
  * d/p/u-aa/lp2079869-* : virt-aa-helper: Avoid duplicate when append rule
    (LP: #2120278)

 -- Hector Cao <email address hidden> Wed, 17 Sep 2025 01:20:45 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 8.0.0-1ubuntu7.13

---------------
libvirt (8.0.0-1ubuntu7.13) jammy; urgency=medium

  * d/p/u/lp-2117467-virdevmapper-device-name-for-targets.patch:
    virdevmapper: Always use device name for finding targets. This ensures
    that all the target devices of a multipath device are added to the
    namespace/cgroup of the guest domain.
    Closes LP: #2117467.

 -- Bhavin Gandhi <email address hidden> Tue, 22 Jul 2025 13:50:20 +0530