== Comment: #0 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:01:18 ==
---Problem Description---
dm-queue-length module is not included in installer/initramfs
On Ubuntu, multipath devices using the 'queue-length' path selector
are non-functional on both the installer and initramfs environments;
because the 'dm-queue-length' kernel module is not included in them.
The multipath-modules.udeb (src:linux) does not include it in the installer,
nor multipath-tools-boot (src:multipath-tools) installs it in the initramfs.
One example is the EMC XtremIO storage, which has 'queue-length' defined as
its path selector in the default multipath configuration, at least on 16.04.
Other products may be affected if they are manually configured to use that
path selector (e.g., via /etc/multipath.conf), and the mere switch of that
might render the system _unbootable_ if booting from multipath, since the
initramfs is affected.
More recently this and another storage changed default path selectors out
of 'queue-length', however, it's virtually possible for any storage system
to be affected, with the described manual configuration change. So, this
change is also desired on for the next stable release, 17.04, and later.
Patches are provided for 16.04 and 17.04.
Error logs:
----------
This error log is seen in the installer, and multipath devices are not
created for such storage system:
...
Mar 14 06:41:54 kernel: [ 0.000000] Kernel command line: tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false --- quiet /install disk-detect/multipath/enable=true
...
Mar 14 06:47:09 main-menu[3743]: INFO: Menu item 'disk-detect' selected
...
Mar 14 06:47:14 anna-install: Installing multipath-udeb
...
Mar 14 06:47:16 kernel: [ 324.968027] device-mapper: table: 252:0: multipath: unknown path selector type
Mar 14 06:47:16 kernel: [ 324.968030] device-mapper: ioctl: error adding target to table
...
Mar 14 06:47:17 kernel: [ 326.484059] device-mapper: table: 252:0: multipath: unknown path selector type
Mar 14 06:47:17 kernel: [ 326.484062] device-mapper: ioctl: error adding target to table
Mar 14 06:47:17 disk-detect: Mar 14 06:47:16 | mpatha: ignoring map
...
Mar 14 06:47:17 disk-detect: Mar 14 06:47:17 | mpathg: ignoring map
Mar 14 06:47:18 disk-detect: No multipath devices detected
And after installing the system in an individual/single path device,
and trying to switch to multipath via multipath-tools-boot, it also
happens in the initramfs:
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=0568fc48-2921-4ba2-b1b5-127c1d5a310e does not exist. Dropping to a shell!
BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
Contact Information = <email address hidden>
---uname output---
4.4.0-31, 4.10.0-11
Machine Type = N/A
---boot type---
CDROM / ISO image
---Install repository type---
CDROM
---Point of failure---
Other failure during installation (stage 1)
== Comment: #1 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:12:30 ==
Patch for 16.04 multipath-tools
== Comment: #2 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:12:56 ==
Patch for 16.04 linux
== Comment: #3 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:13:18 ==
Patch for 17.04 multipath-tools
== Comment: #4 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:16:27 ==
Patch verification on 16.04:
---------------------------
Please assign this bug to @cyphermox for the multipath-tools components, at least, as a suggestion.
He's handled most of the multipath-tools related bugs/patches we've been contributing with.
Thank you.
== Comment: #10 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:40:17 ==
It's possible to reproduce this problem in a qemu-kvm guest,
with an emulated disk: force the queue-length path selector.
== Comment: #0 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:01:18 ==
---Problem Description---
dm-queue-length module is not included in installer/initramfs
On Ubuntu, multipath devices using the 'queue-length' path selector
are non-functional on both the installer and initramfs environments;
because the 'dm-queue-length' kernel module is not included in them.
The multipath- modules. udeb (src:linux) does not include it in the installer, tools-boot (src:multipath- tools) installs it in the initramfs.
nor multipath-
One example is the EMC XtremIO storage, which has 'queue-length' defined as
its path selector in the default multipath configuration, at least on 16.04.
Other products may be affected if they are manually configured to use that conf), and the mere switch of that
path selector (e.g., via /etc/multipath.
might render the system _unbootable_ if booting from multipath, since the
initramfs is affected.
More recently this and another storage changed default path selectors out
of 'queue-length', however, it's virtually possible for any storage system
to be affected, with the described manual configuration change. So, this
change is also desired on for the next stable release, 17.04, and later.
Patches are provided for 16.04 and 17.04.
Error logs:
----------
This error log is seen in the installer, and multipath devices are not
created for such storage system:
... language- pack-patterns= pkgsel/ install- language- support= false --- quiet /install disk-detect/ multipath/ enable= true
Mar 14 06:41:54 kernel: [ 0.000000] Kernel command line: tasks=standard pkgsel/
...
Mar 14 06:47:09 main-menu[3743]: INFO: Menu item 'disk-detect' selected
...
Mar 14 06:47:14 anna-install: Installing multipath-udeb
...
Mar 14 06:47:16 kernel: [ 324.968027] device-mapper: table: 252:0: multipath: unknown path selector type
Mar 14 06:47:16 kernel: [ 324.968030] device-mapper: ioctl: error adding target to table
...
Mar 14 06:47:17 kernel: [ 326.484059] device-mapper: table: 252:0: multipath: unknown path selector type
Mar 14 06:47:17 kernel: [ 326.484062] device-mapper: ioctl: error adding target to table
Mar 14 06:47:17 disk-detect: Mar 14 06:47:16 | mpatha: ignoring map
...
Mar 14 06:47:17 disk-detect: Mar 14 06:47:17 | mpathg: ignoring map
Mar 14 06:47:18 disk-detect: No multipath devices detected
And after installing the system in an individual/single path device, tools-boot, it also
and trying to switch to multipath via multipath-
happens in the initramfs:
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. local-premount ... Begin: Waiting for udev to settle (multipath) ... done. local-block ... done. local-block ... done. local-block ... done. local-block ... done.
Begin: Running /scripts/
[ 89.709480] device-mapper: table: 252:0: multipath: unknown path selector type
[ 89.709537] device-mapper: ioctl: error adding target to table
[ 89.756251] device-mapper: table: 252:0: multipath: unknown path selector type
[ 89.837132] device-mapper: ioctl: error adding target to table
[ 89.912346] device-mapper: table: 252:0: multipath: unknown path selector type
[ 90.001719] device-mapper: ioctl: error adding target to table
[ 90.300203] device-mapper: table: 252:0: multipath: unknown path selector type
[ 90.303521] device-mapper: ioctl: error adding target to table
Begin: Running /scripts/
Begin: Running /scripts/
Begin: Running /scripts/
...
Begin: Running /scripts/
done.
Gave up waiting for root device. Common problems: 2921-4ba2- b1b5-127c1d5a31 0e does not exist. Dropping to a shell!
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=0568fc48-
BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
Contact Information = <email address hidden>
---uname output---
4.4.0-31, 4.10.0-11
Machine Type = N/A
---boot type---
CDROM / ISO image
---Install repository type---
CDROM
---Point of failure---
Other failure during installation (stage 1)
== Comment: #1 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:12:30 ==
Patch for 16.04 multipath-tools
== Comment: #2 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:12:56 ==
Patch for 16.04 linux
== Comment: #3 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:13:18 ==
Patch for 17.04 multipath-tools
== Comment: #4 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:16:27 == ------- ------- ------
Patch verification on 16.04:
-------
multipath-tools source package:
# sed -n '/vendor. *XtremIO/ ,/selector/ p' libmultipath/ hwtable. c
.vendor = "XtremIO",
.product = "XtremApp",
.features = DEFAULT_FEATURES,
.hwhandler = DEFAULT_HWHANDLER,
.selector = "queue-length 0",
- initramfs:
---------
Before (module not included in initramfs):
# dpkg -s multipath- tools-boot | grep ^Version: 656f8865- 5ubuntu2. 4
Version: 0.5.0+git1.
# lsinitramfs /boot/initrd.img | grep dm-queue-length
#
After (module is included in initramfs):
# dpkg -i multipath- tools*. deb kpartx*.deb
# dpkg -s multipath- tools-boot | grep ^Version: 656f8865- 5ubuntu2. 4dmqueuelength1
Version: 0.5.0+git1.
# lsinitramfs /boot/initrd.img | grep dm-queue-length 4.4.0-66- generic/ kernel/ drivers/ md/dm-queue- length. ko
lib/modules/
- kernel udeb:
-----------
# dpkg-deb -c multipath- modules- 4.4.0-66- generic- di_4.4. 0-66.87dmqueuel ength1_ ppc64el. udeb | grep dm-queue-length 4.4.0-66- generic/ kernel/ drivers/ md/dm-queue- length. ko
-rw-r--r-- root/root 12174 2017-03-14 17:17 ./lib/modules/
Patch verification on 17.04: ------- ------- ------
-------
Before:
# dpkg -s multipath- tools-boot | grep ^Version:
Version: 0.6.4-3ubuntu1
# lsinitramfs /boot/initrd.img | grep dm-queue-length
#
After:
# dpkg -i multipath- tools*. deb kpartx*.deb
# dpkg -s multipath- tools-boot | grep ^Version: mqueuelength1
Version: 0.6.4-3ubuntu1d
# lsinitramfs /boot/initrd.img | grep dm-queue-length 4.10.0- 11-generic/ kernel/ drivers/ md/dm-queue- length. ko
lib/modules/
- kernel udeb:
-----------
# dpkg-deb -c multipath- modules- 4.10.0- 11-generic- di_4.10. 0-11.13dmqueuel ength1_ ppc64el. udeb | grep dm-queue-length 4.10.0- 11-generic/ kernel/ drivers/ md/dm-queue- length. ko
-rw-r--r-- root/root 13128 2017-03-15 11:14 ./lib/modules/
== Comment: #5 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:20:14 ==
Patch for 17.04 linux
== Comment: #9 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:24:40 ==
@taco-screen-team
Please assign this bug to @cyphermox for the multipath-tools components, at least, as a suggestion.
He's handled most of the multipath-tools related bugs/patches we've been contributing with.
Thank you.
== Comment: #10 - Mauricio Faria De Oliveira <email address hidden> - 2017-03-15 09:40:17 ==
It's possible to reproduce this problem in a qemu-kvm guest,
with an emulated disk: force the queue-length path selector.
# name=mfo-1704 lib/libvirt/ images/ $name.qcow2 lib/libvirt/ images/ zesty-server- ppc64el. iso.2017- 03-14
# disk=/var/
# iso=/var/
# qemu-img create -f qcow2 $disk 128g
# virt-install \ 1,cores= 1,threads= 8 \ model=virtio- scsi \ qcow2,path= $disk \ cdrom,readonly= true,path= $iso \ virbr0, model=virtio
--name $name \
--cdrom $iso \
--vcpus 8,sockets=
--memory 8192 \
--controller type=scsi,
--disk format=
--disk device=
--network bridge=
Before the disk-detection stage (e.g., set-up users and passwords),
Go Back, Execute a shell, and set up multipath.conf, and resume install:
~ # cat <<EOF >/etc/multipath .conf friendly_ names yes
defaults {
path_selector "queue-length 0"
user_
find_multipaths no
}
EOF
~ # exit