vgscan --mknodes creates block device multipath nodes in /dev/mapper

[Ryan Harper]

Description: Ubuntu 20.04 LTS
Release: 20.04

# apt-cache policy multipath-tools
multipath-tools:
  Installed: 0.8.3-1ubuntu2
  Candidate: 0.8.3-1ubuntu2
  Version table:
 *** 0.8.3-1ubuntu2 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/main s390x Packages
        100 /var/lib/dpkg/status

/dev/mapper/mpatha is symlink which points to ../../dm-X device

/dev/mapper/mpath is block device

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: multipath-tools 0.8.3-1ubuntu2
ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
Uname: Linux 5.4.0-26-generic s390x
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu27
Architecture: s390x
CasperMD5CheckResult: pass
CasperVersion: 1.445
Date: Thu Apr 23 17:32:12 2020
LiveMediaBuild: Ubuntu-Server 20.04 LTS "Focal Fossa" - Release s390x (20200423.1)
ProcEnviron:
 TERM=screen.xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: multipath-tools
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.multipath.conf:
 defaults {
     user_friendly_names yes
     skip_kpartx no
     verbosity 4
 }
mtime.conffile..etc.multipath.conf: 2020-04-23T15:47:28.659850

[Michael Hudson-Doyle (mwhudson)]

Actually it seems this was caused by subiquity running vgscan --mknodes during block probing:

root@ubuntu-server:/# rm -rf /dev/mapper/mpath*
root@ubuntu-server:/# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Apr 24 05:57 control
root@ubuntu-server:/# vgscan --mknodes
root@ubuntu-server:/# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Apr 24 05:57 control
brw-rw---- 1 root disk 253, 0 Apr 27 21:16 mpatha
brw-rw---- 1 root disk 253, 1 Apr 27 21:16 mpatha-part1

retargeting bug...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ryan Harper (raharper)]

This patch configures the dm_task to use the DM_UDEV_DISABLE_LIBRARY_FALLBACK udev flag to prevent calls to dm_mknodes() from using the library fallback code.

Another possible fix would be to obtain a different return code from vgscan.c where it does not detect any VGs and then calls the mknode path due to the argument passed to the cli

    maxret = process_each_vg(cmd, argc, argv, NULL, NULL, 0, 0, NULL, &_vgscan_single);

    /* at this point, this could return some indication if vgs were found
       such that we could skip the vgmknodes() if no vgs were found */

    if (arg_is_set(cmd, mknodes_ARG)) {
        ret = vgmknodes(cmd, argc, argv);
        if (ret > maxret)
            maxret = ret;
    }

[Ryan Harper (raharper)]

Note, the whitespace damage in the patch is not needed, just my text client cleaning up trailing whitespace.

With this patch applied, vgscan --mknodes no longer creates block devices in /dev/mapper

root@ubuntu:/dev/mapper# ls -al
total 0
drwxr-xr-x 2 root root 120 Apr 28 16:31 .
drwxr-xr-x 17 root root 3800 Apr 28 15:25 ..
crw------- 1 root root 10, 236 Apr 28 15:22 control
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha -> ../dm-0
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha-part1 -> ../dm-1
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha-part2 -> ../dm-2
root@ubuntu:/dev/mapper#
root@ubuntu:/dev/mapper#
root@ubuntu:/dev/mapper# ls -al
total 0
drwxr-xr-x 2 root root 120 Apr 28 16:31 .
drwxr-xr-x 17 root root 3800 Apr 28 15:25 ..
crw------- 1 root root 10, 236 Apr 28 15:22 control
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha -> ../dm-0
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha-part1 -> ../dm-1
lrwxrwxrwx 1 root root 7 Apr 28 16:31 mpatha-part2 -> ../dm-2
root@ubuntu:/dev/mapper# rm -rf mpatha*
root@ubuntu:/dev/mapper# vgscan --mknodes
root@ubuntu:/dev/mapper# ls -al
total 0
drwxr-xr-x 2 root root 60 Apr 28 16:40 .
drwxr-xr-x 17 root root 3800 Apr 28 15:25 ..
crw------- 1 root root 10, 236 Apr 28 15:22 control

The relevant output:

16:40:35.179504 vgscan[55466] toollib.c:2288 No volume groups found.
16:40:35.179806 vgscan[55466] device_mapper/libdm-common.c:2205 WARK: setting a cookie
16:40:35.180137 vgscan[55466] device_mapper/libdm-common.c:2565 Udev cookie 0xd4d8d17 (semid 32786) created
16:40:35.180451 vgscan[55466] device_mapper/libdm-common.c:2585 Udev cookie 0xd4d8d17 (semid 32786) incremented to 1
16:40:35.180762 vgscan[55466] device_mapper/libdm-common.c:2457 Udev cookie 0xd4d8d17 (semid 32786) incremented to 2
16:40:35.181082 vgscan[55466] device_mapper/libdm-common.c:2690 Udev cookie 0xd4d8d17 (semid 32786) assigned to MKNODES task(15) with flags DISABLE_LIBRARY_FALLBACK (0x20)
16:40:35.181398 vgscan[55466] device_mapper/libdm-common.c:2709 WARK: got my cookie, return 1
16:40:35.181714 vgscan[55466] device_mapper/libdm-common.c:2214 WARK: running dmt with DM-DEV_DISABLE_LIBRARY_FALLBACK flag
16:40:35.182048 vgscan[55466] device_mapper/ioctl/libdm-iface.c:1853 dm names [ opencount flush ] [16384] (*1)
16:40:35.182379 vgscan[55466] device_mapper/ioctl/libdm-iface.c:1853 dm mknodes mpatha [ noopencount flush ] [16384] (*1)
16:40:35.182687 vgscan[55466] device_mapper/libdm-common.c:1484 mpatha: Stacking NODE_ADD (253,0) 0:6 0660 [trust_udev]
16:40:35.183004 vgscan[55466] device_mapper/ioctl/libdm-iface.c:1853 dm mknodes mpatha-part2 [ noopencount flush ] [16384] (*1)
16:40:35.183341 vgscan[55466] device_mapper/libdm-common.c:1484 mpatha-part2: Stacking NODE_ADD (253,2) 0:6 0660 [trust_udev]
16:40:35.183648 vgscan[55466] device_mapper/ioctl/libdm-iface.c:1853 dm mknodes mpatha-part1 [ noopencount flush ] [16384] (*1)
16:40:35.183968 vgscan[55466] device_mapper/libdm-common.c:1484 mpatha-part1: Stacking NODE_ADD (253,1) 0:6 0660 [trust_udev]
16:40:35.184275 vgscan[55466] device_mapper/libdm-common.c:2218 WARK: running out path
16:40:...

[Michael Hudson-Doyle (mwhudson)]

FWIW I reported this upstream at https://bugzilla.redhat.com/show_bug.cgi?id=1828617, you might want to attach the patch there too?

[Ryan Harper (raharper)]

I don't think this is the right fix. We need to test that it doesn't prevent vgscan from finding and creating /dev/vg/lv device files outside of of multipath.

The real fix involves:

  1) omitting the call to vgmknodes if there are no volume groups present
  2) if vgnodes are present, device_mapper/libdm-common.c should determine if DM_DISABLE_LIBRARY_FALLBACK should be set in the same way that dmsetup.c does.

I'd much prefer upstream to produce a proper fix.

[Ryan Harper (raharper)]

Probert PR to drop --mknodes:
    https://github.com/canonical/probert/pull/88
Curtin MP to drop --mknodes
    https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/383141

[Frank Heimes (fheimes)]

Based on the PR/MP in comment #8 and the IRC chat about this bug,
I'm updating the status to Fix Committed.

[Dimitri John Ledkov (xnox)]

lvm2 is buggy, but we're just gonna tip toe around it.

probert/curtin/subiquity are fixed, and lvm2 can keep it's trap open for the next software to walk into it.