NVMe drives aren't discovered by Fuel

[Artem Panchenko]

Fuel 8.0 build #429, baremetal servers with NVMe SSD drives are discovered by nailgun-agent, but their disks aren't detected (see screenshot):

D, [2016-01-19T15:06:19.346297 #729] DEBUG -- : Trying to get list of physical devices
D, [2016-01-19T15:06:19.346515 #729] DEBUG -- : Getting udev properties for device: nvme0n1
D, [2016-01-19T15:06:19.350220 #729] DEBUG -- : Device nvme0n1 udev properties: {"DEVLINKS"=>"/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000", "DEVNAME"=>"/dev/nvme0n1", "DEVPATH"=>"/devices/pci0000:00/0000:00:02.0/0000:02:00.0/block/nvme0n1", "DEVTYPE"=>"disk", "ID_MODEL"=>"INTEL_SSDPEDMD40", "ID_MODEL_ENC"=>"INTEL\\x20SSDPEDMD40", "ID_REVISION"=>"0131", "ID_SCSI"=>"1", "ID_SCSI_SERIAL"=>"CVFT516600MC400BGN", "ID_SERIAL"=>"365cd2e4080864356494e000000010000", "ID_SERIAL_SHORT"=>"65cd2e4080864356494e000000010000", "ID_TYPE"=>"disk", "ID_VENDOR"=>"NVMe", "ID_VENDOR_ENC"=>"NVMe\\x20\\x20\\x20\\x20", "ID_WWN"=>"0x65cd2e4080864356", "ID_WWN_VENDOR_EXTENSION"=>"0x494e000000010000", "ID_WWN_WITH_EXTENSION"=>"0x65cd2e4080864356494e000000010000", "MAJOR"=>"259", "MINOR"=>"0", "SUBSYSTEM"=>"block", "TAGS"=>":systemd:", "USEC_INITIALIZED"=>"814121"}
D, [2016-01-19T15:06:19.350271 #729] DEBUG -- : Trying to find out if device nvme0n1 is removable or not
D, [2016-01-19T15:06:19.350361 #729] DEBUG -- : Device nvme0n1 removable parameter: "0"
D, [2016-01-19T15:06:19.350423 #729] DEBUG -- : Getting udev properties for device: nvme1n1
D, [2016-01-19T15:06:19.353952 #729] DEBUG -- : Device nvme1n1 udev properties: {"DEVLINKS"=>"/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000", "DEVNAME"=>"/dev/nvme1n1", "DEVPATH"=>"/devices/pci0000:00/0000:00:03.0/0000:03:00.0/block/nvme1n1", "DEVTYPE"=>"disk", "ID_MODEL"=>"INTEL_SSDPEDMD40", "ID_MODEL_ENC"=>"INTEL\\x20SSDPEDMD40", "ID_REVISION"=>"0131", "ID_SCSI"=>"1", "ID_SCSI_SERIAL"=>"CVFT5073009M400BGN", "ID_SERIAL"=>"365cd2e4080864356494e000000010000", "ID_SERIAL_SHORT"=>"65cd2e4080864356494e000000010000", "ID_TYPE"=>"disk", "ID_VENDOR"=>"NVMe", "ID_VENDOR_ENC"=>"NVMe\\x20\\x20\\x20\\x20", "ID_WWN"=>"0x65cd2e4080864356", "ID_WWN_VENDOR_EXTENSION"=>"0x494e000000010000", "ID_WWN_WITH_EXTENSION"=>"0x65cd2e4080864356494e000000010000", "MAJOR"=>"259", "MINOR"=>"1", "SUBSYSTEM"=>"block", "TAGS"=>":systemd:", "USEC_INITIALIZED"=>"916983"}
D, [2016-01-19T15:06:19.354002 #729] DEBUG -- : Trying to find out if device nvme1n1 is removable or not
D, [2016-01-19T15:06:19.354088 #729] DEBUG -- : Device nvme1n1 removable parameter: "0"
D, [2016-01-19T15:06:19.354123 #729] DEBUG -- : Final list of physical devices is: []
D, [2016-01-19T15:06:19.354177 #729] DEBUG -- : Found block device: nvme1n1
D, [2016-01-19T15:06:19.354210 #729] DEBUG -- : Block device info: {"size"=>"781422768", "removable"=>"0", "vendor"=>"0x8086"}
D, [2016-01-19T15:06:19.354230 #729] DEBUG -- : Trying to get list of physical devices
D, [2016-01-19T15:06:19.354314 #729] DEBUG -- : Getting udev properties for device: nvme0n1
D, [2016-01-19T15:06:19.357448 #729] DEBUG -- : Device nvme0n1 udev properties: {"DEVLINKS"=>"/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000", "DEVNAME"=>"/dev/nvme0n1", "DEVPATH"=>"/devices/pci0000:00/0000:00:02.0/0000:02:00.0/block/nvme0n1", "DEVTYPE"=>"disk", "ID_MODEL"=>"INTEL_SSDPEDMD40", "ID_MODEL_ENC"=>"INTEL\\x20SSDPEDMD40", "ID_REVISION"=>"0131", "ID_SCSI"=>"1", "ID_SCSI_SERIAL"=>"CVFT516600MC400BGN", "ID_SERIAL"=>"365cd2e4080864356494e000000010000", "ID_SERIAL_SHORT"=>"65cd2e4080864356494e000000010000", "ID_TYPE"=>"disk", "ID_VENDOR"=>"NVMe", "ID_VENDOR_ENC"=>"NVMe\\x20\\x20\\x20\\x20", "ID_WWN"=>"0x65cd2e4080864356", "ID_WWN_VENDOR_EXTENSION"=>"0x494e000000010000", "ID_WWN_WITH_EXTENSION"=>"0x65cd2e4080864356494e000000010000", "MAJOR"=>"259", "MINOR"=>"0", "SUBSYSTEM"=>"block", "TAGS"=>":systemd:", "USEC_INITIALIZED"=>"814121"}
D, [2016-01-19T15:06:19.357502 #729] DEBUG -- : Trying to find out if device nvme0n1 is removable or not
D, [2016-01-19T15:06:19.357629 #729] DEBUG -- : Device nvme0n1 removable parameter: "0"
D, [2016-01-19T15:06:19.357673 #729] DEBUG -- : Getting udev properties for device: nvme1n1
D, [2016-01-19T15:06:19.367494 #729] DEBUG -- : Device nvme1n1 udev properties: {"DEVLINKS"=>"/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000", "DEVNAME"=>"/dev/nvme1n1", "DEVPATH"=>"/devices/pci0000:00/0000:00:03.0/0000:03:00.0/block/nvme1n1", "DEVTYPE"=>"disk", "ID_MODEL"=>"INTEL_SSDPEDMD40", "ID_MODEL_ENC"=>"INTEL\\x20SSDPEDMD40", "ID_REVISION"=>"0131", "ID_SCSI"=>"1", "ID_SCSI_SERIAL"=>"CVFT5073009M400BGN", "ID_SERIAL"=>"365cd2e4080864356494e000000010000", "ID_SERIAL_SHORT"=>"65cd2e4080864356494e000000010000", "ID_TYPE"=>"disk", "ID_VENDOR"=>"NVMe", "ID_VENDOR_ENC"=>"NVMe\\x20\\x20\\x20\\x20", "ID_WWN"=>"0x65cd2e4080864356", "ID_WWN_VENDOR_EXTENSION"=>"0x494e000000010000", "ID_WWN_WITH_EXTENSION"=>"0x65cd2e4080864356494e000000010000", "MAJOR"=>"259", "MINOR"=>"1", "SUBSYSTEM"=>"block", "TAGS"=>":systemd:", "USEC_INITIALIZED"=>"916983"}
D, [2016-01-19T15:06:19.367536 #729] DEBUG -- : Trying to find out if device nvme1n1 is removable or not
D, [2016-01-19T15:06:19.367624 #729] DEBUG -- : Device nvme1n1 removable parameter: "0"
D, [2016-01-19T15:06:19.367655 #729] DEBUG -- : Final list of physical devices is: []
D, [2016-01-19T15:06:19.367677 #729] DEBUG -- : Detailed meta disks: []

Bootstrap OS (Ubuntu) recognizes the drives:

[root@bootstrap ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 372.6G 0 disk
nvme1n1 259:1 0 372.6G 0 disk
[root@bootstrap ~]# fdisk -l

Disk /dev/nvme0n1: 400.1 GB, 400088457216 bytes, 781422768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/nvme1n1: 400.1 GB, 400088457216 bytes, 781422768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

You can find output of `udevadm info -n /dev/nvme{0,1} -a` command in attachments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/270099

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/270109

[Alexander Gordeev (a-gordeev)]

Regarding 7.0 updates:
1) centos flavoured bootstrap image isn't affected. NVMe has major number code of 253.

2) 7.0 was the first release of Fuel where ubuntu flavoured bootstrap image feature appeared under experimental status. Unfortunately, affected. AFAIK, experimental features are not supported in terms of updates. But i might be wrong.

Kindly asking maintenance team to take a decision.

[Sergii Rizvan (srizvan)]

We are going to close this bug for 7.0 as Won't Fix because centos flavoured bootstrap image isn't affected and ubuntu flavored bootstrap image is an experimental feature in this release. Due to Maintenance policy we don't fix bugs in experimental features.

P.S. There is another bug in Fuel, where deployment fails with multiple NVME disks: https://bugs.launchpad.net/fuel/+bug/1503987. It's already fixed in 8.0 and 7.0.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270099
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=18289c69ffcb9a34208e51dd6a30de5178928e05
Submitter: Jenkins
Branch: master

commit 18289c69ffcb9a34208e51dd6a30de5178928e05
Author: Alexander Gordeev <email address hidden>
Date: Wed Jan 20 13:01:26 2016 +0300

    Stop filtering NVME devices by major number

    Apparently, nailgun-agent filters out NVME devices due to
    major number mismatch.

    This wasn't an issue with kernel shipped with CentOS 6.x
    where major number was 253.

    For new kernel version valid major number is 259.

    Change-Id: Idc572832b3a5650496439c32c0addcca4d759378
    Closes-Bug: #1536055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270109
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=b2bb466fd5bd92da614cdbd819d6999c510ebfb1
Submitter: Jenkins
Branch: stable/8.0

commit b2bb466fd5bd92da614cdbd819d6999c510ebfb1
Author: Alexander Gordeev <email address hidden>
Date: Wed Jan 20 13:01:26 2016 +0300

    Stop filtering NVME devices by major number

    Apparently, nailgun-agent filters out NVME devices due to
    major number mismatch.

    This wasn't an issue with kernel shipped with CentOS 6.x
    where major number was 253.

    For new kernel version valid major number is 259.

    Change-Id: Idc572832b3a5650496439c32c0addcca4d759378
    Closes-Bug: #1536055

[Artem Panchenko (apanchenko-8)]

verified

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "506"
  build_id: "506"
  fuel-nailgun_sha: "8e954abd70ef0083109f34289de2553dcda544d4"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "ec7e212972ead554f21b52b9e165156665f659df"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "351d568fa3b3e4dd062054b91d766aa54d379867"
  fuelmenu_sha: "234cb4cbb30fbd2df00f388c28f31606d9cae15f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "94507c5e4dad6d8cfbd8f5d41aa8389d5335990a"