Nova cannot use the multipath path for creating VMs

[Serhii Ovsianikov]

MOS 7.0

After updating the "multipath-tools" package nova cannot use the multipath path for creating VMs.

In the libvirt.xml file of the instance we can see not multipath path <source dev>:
<source dev="/dev/disk/by-path/pci-0000:09:00.0-fc-0x21110002ac015b33-lun-1"/>

The multipath path should appear as:
<source dev="/dev/mapper/360002ac000000000000000a200015b33"/>

In the nova-compute.log file we can see the following warnings:

2016-05-03 12:42:46.503 8299 WARNING nova.storage.linuxscsi [req-27431375-e6e1-4ee8-9230-1edb57a649b7 0e5e134325e145d5b5b60b6d24be5aa6 cf2c57ebfaff44a9b5ec9c51164b5e2d - - -] Couldn't find multipath device /dev/mapper/May

2016-04-07 09:34:24.286 9549 WARNING nova.storage.linuxscsi [req-406022ef-1135-4f36-9ad9-615331ad529a 0e5e134325e145d5b5b60b6d24be5aa6 cf2c57ebfaff44a9b5ec9c51164b5e2d - - -] Couldn't find multipath device /dev/mapper/Apr

The output of "multipath -l" command is
# multipath -l /dev/sdc
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
May 13 12:53:50 | /lib/udev/scsi_id exitted with 1
360002ac0000000000000028900015b33 dm-3 3PARdata,VV
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  |- 1:0:0:0 sdc 8:32 active undef running
  |- 1:0:1:0 sdb 8:16 active undef running
  |- 2:0:1:0 sdd 8:48 active undef running
  `- 2:0:0:0 sde 8:64 active undef running

In the old version of multipath-tools, the output was
# multipath -l /dev/sdh
360002ac0000000000000013d00015b33 dm-3 3PARdata,VV
size=19G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  |- 1:0:0:0 sdb 8:16 active undef running
  |- 1:0:0:1 sdc 8:32 active undef running
  |- 1:0:0:2 sdd 8:48 active undef running
  |- 1:0:0:3 sde 8:64 active undef running
  |- 1:0:1:0 sdg 8:96 active undef running
  |- 1:0:1:1 sdh 8:112 active undef running
  |- 1:0:1:2 sdi 8:128 active undef running
  |- 1:0:1:3 sdj 8:144 active undef running
  |- 2:0:0:0 sdl 8:176 active undef running
  |- 2:0:0:1 sdm 8:192 active undef running
  |- 2:0:0:2 sdn 8:208 active undef running
  |- 2:0:0:3 sdo 8:224 active undef running
  |- 2:0:1:0 sdq 65:0 active undef running
  |- 2:0:1:1 sdr 65:16 active undef running
  |- 2:0:1:2 sds 65:32 active undef running
  `- 2:0:1:3 sdt 65:48 active undef running

[Serhii Ovsianikov (sovsianikov)]

In addition: MOS 7.0

[Roman Podoliaka (rpodolyaka)]

This is fixed in Liberty in os-brick:

https://github.com/openstack/os-brick/blob/stable/liberty/os_brick/initiator/linuxscsi.py#L270-L271

so MOS 8.0 is not affected.

There is a patch on review for stable/kilo:

https://review.openstack.org/#/c/258191/

which will never be merged (Kilo is reaching EOL).

The issue itself does not seem to be of High importance to me - it's not clear what conditions need to be met for it to be reproduced. Medium seems to be the right priority. But we'll let MOS Maintenance to decide, whether we should merge this into the stable branch or not.

The patch does not look dangerous, but neither it's reliable: there might be some additional warnings we have not seen yet, which will not be parsed correctly. I'd rather we find a way to reproduce this and similar issues before we merge this change.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Bjoern Teipel <email address hidden>
Review: https://review.fuel-infra.org/20684

[Serhii Ovsianikov (sovsianikov)]

In version 7.9, multipath is trying to scan "nbd" devices by "default", now and this is producing this error message "/lib/udev/scsi_id exitted with 1".

May 17 08:38:47 | nbd4: not found in pathvec
May 17 08:38:47 | nbd4: mask = 0x31
May 17 08:38:47 | nbd4: dev_t = 43:64
May 17 08:38:47 | nbd4: size = 0
May 17 08:38:47 | nbd4: getuid = /lib/udev/scsi_id --whitelisted --device=/dev/%n (internal default)
May 17 08:38:47 | /lib/udev/scsi_id exitted with 1
May 17 08:38:47 | error calling out /lib/udev/scsi_id --whitelisted --device=/dev/nbd4

The multipath-tools of version "amd 0.4.9-3ubuntu7.4" works fine

Update from "multipath-tools:amd64 0.4.9-3ubuntu7.4" to "multipath-tools:amd64 0.4.9-3ubuntu7.9" and multipath is scanning now for nbd devices by default.

The latest version "multipath-tools:amd64 0.4.9-3ubuntu7.11" is also scanning nbd devices by default.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20684
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 97eca1abe051691e27b198c022c2e17888d3a90d
Author: Bjoern Teipel <email address hidden>
Date: Tue May 31 12:15:37 2016

libvirt: ignore error messages from multipath command output before parsing

This fixes an issue in _get_multipath_device_name() that fails to
parse the output from 'multipath' command when the stdout contains
error messages in addition to the expected output. As a result
libvirt will be instructed to attach wrong, not existing devices names,
to the instance. This behavior will result in instances not being able
to boot anymore unless manually fixed.

Co-Authored-By: Tomoki Sekiyama <email address hidden>
Closes-Bug: #1581575

Before os-brick (which we started to use in Liberty) we had multiple
different wrappers around multipath call, that's why we also need to
modify nova.storage.linuxscsi module to introduce the very same
filtering and keep the changes to minimum (as it's a backport to
a stable branch).

(cherry-picked from 39c37abf8be503c6f199ccd58903705d143be22c)
Change-Id: I6bdcbeecb1330ed6b5c8bbe928ecfc7b3062aa10

[Dmitry (dtsapikov)]

Verified on 7.0+mu4