In Fuel 7.0 with using more than 1 NMVe disk the deployment fails.
Fuel agens comes with an error
Disk not found: <diskname> , where diskname is nvme1n1 or nvme2n1
The error comes from the fuction '_disk_dev' in the fuel_agent / drivers / nailgun.py
So i've been digging arround and i found the cause of the error
In the bootstrap image based on CentOS 'udevadm' in not working correctly. It returns the id of the disk incorrectly. Id's of two nvme disks are the same.
Here is an example of a server with 3 disks, 1 hardware raid and two nvme disks. (which are on the pci bus)
[root@bootstrap ~]# udevadm info --export-db | grep /dev/disk
E: DEVLINKS=/dev/block/253:0 /dev/disk/by-path/pci-0000:04:00.0 /dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000
E: DEVLINKS=/dev/block/253:64 /dev/disk/by-path/pci-0000:05:00.0 /dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000
E: DEVLINKS=/dev/block/8:0 /dev/disk/by-id/scsi-3600304801c1e8f001d5cb9c602dff27b /dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0 /dev/disk/by-id/wwn-0x600304801c1e8f001d5cb9c602dff27b
This causes the function 'match_device' in fuel_agent / drivers /nailgun.py to return two block devices (for example an array ['/dev/nvme0n1','/dev/nvme1n1']).
Which in the function '_dev_disk' causes the var 'len(found) > 1' and throws the Disk not found error.
a temponary work arroud i used in (and i then succesfully deployed an env) is to modify the function '_dev_disk' to this
def _disk_dev(self, ks_disk):
# first we try to find a device that matches ks_disk
# comparing by-id and by-path links
matched = [hu_disk['device'] for hu_disk in self.hu_disks
if match_device(hu_disk, ks_disk)]
# if we can not find a device by its by-id and by-path links
# we try to find a device by its name
fallback = [hu_disk['device'] for hu_disk in self.hu_disks
if '/dev/%s' % ks_disk['name'] == hu_disk['device']]
# fix for centOS since udevadm is not reporting correctly
# choose in case of nvme disks only fallback method
if any(f.find('nvme') > 0 for f in matched):
matched = False
# end of fix
found = matched or fallback
if not found or len(found) > 1:
raise errors.DiskNotFoundError(
'Disk not found: %s ' % ks_disk['name'])
so in case of nvme, only use the fallback method.
Here is some data to play with:
hu_disks
[{'bspec': {'alignoff': '0',
'iomin': '512',
'ioopt': '0',
'maxsect': '256',
'pbsz': '512',
'ra': '256',
'ro': '0',
'size64': '800166076416',
'ss': '512',
'sz': '1562824368'},
'device': '/dev/nvme0n1',
'espec': {'removable': '0', 'vendor': '0x8086'},
'uspec': {'DEVLINKS': ['/dev/block/253:0',
'/dev/disk/by-path/pci-0000:04:00.0',
'/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000'],
'DEVNAME': '/dev/nvme0n1',
'DEVPATH': '/devices/pci0000:00/0000:00:03.0/0000:04:00.0/block/nvme0n1',
'DEVTYPE': 'disk',
'ID_MODEL': 'INTEL_SSDPE2ME80',
'ID_SERIAL_SHORT': '65cd2e4080864356494e000000010000',
'ID_VENDOR': 'NVMe',
'ID_WWN': '0x65cd2e4080864356',
'MAJOR': '253',
'MINOR': '0'}},
{'bspec': {'alignoff': '0',
'iomin': '512',
'ioopt': '0',
'maxsect': '256',
'pbsz': '512',
'ra': '256',
'ro': '0',
'size64': '800166076416',
'ss': '512',
'sz': '1562824368'},
'device': '/dev/nvme1n1',
'espec': {'removable': '0', 'vendor': '0x8086'},
'uspec': {'DEVLINKS': ['/dev/block/253:64',
'/dev/disk/by-path/pci-0000:05:00.0',
'/dev/disk/by-id/wwn-0x65cd2e4080864356494e000000010000'],
'DEVNAME': '/dev/nvme1n1',
'DEVPATH': '/devices/pci0000:00/0000:00:03.1/0000:05:00.0/block/nvme1n1',
'DEVTYPE': 'disk',
'ID_MODEL': 'INTEL_SSDPE2ME80',
'ID_SERIAL_SHORT': '65cd2e4080864356494e000000010000',
'ID_VENDOR': 'NVMe',
'ID_WWN': '0x65cd2e4080864356',
'MAJOR': '253',
'MINOR': '64'}},
{'bspec': {'alignoff': '0',
'iomin': '4096',
'ioopt': '0',
'maxsect': '560',
'pbsz': '4096',
'ra': '256',
'ro': '0',
'size64': '79456894976',
'ss': '512',
'sz': '155189248'},
'device': '/dev/sda',
'espec': {'removable': '0',
'state': 'running',
'timeout': '90',
'vendor': 'LSI'},
'uspec': {'DEVLINKS': ['/dev/block/8:0',
'/dev/disk/by-id/scsi-3600304801c1e8f001d5cb9c602dff27b',
'/dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0',
'/dev/disk/by-id/wwn-0x600304801c1e8f001d5cb9c602dff27b'],
'DEVNAME': '/dev/sda',
'DEVPATH': '/devices/pci0000:80/0000:80:01.0/0000:82:00.0/host10/target10:2:0/10:2:0:0/block/sda',
'DEVTYPE': 'disk',
'ID_BUS': 'scsi',
'ID_MODEL': 'SMC3108',
'ID_SERIAL_SHORT': '600304801c1e8f001d5cb9c602dff27b',
'ID_VENDOR': 'LSI',
'ID_WWN': '0x600304801c1e8f00',
'MAJOR': '8',
'MINOR': '0'}}]
ks_disk
{'extra': ['disk/by-id/wwn-0x65cd2e4080864356494e000000010000'],
'free_space': 762469,
'id': 'disk/by-path/pci-0000:05:00.0',
'name': 'nvme0n1',
'size': 763097,
'type': 'disk',
'volumes': [{'size': 300, 'type': 'boot'},
{'file_system': 'ext2',
'mount': '/boot',
'name': 'Boot',
'size': 200,
'type': 'raid'},
{'size': 0, 'type': 'lvm_meta_pool'},
{'lvm_meta_size': 64, 'size': 55360, 'type': 'pv', 'vg': 'os'},
{'lvm_meta_size': 64, 'size': 707237, 'type': 'pv', 'vg': 'vm'}]}
Hi, Atze de Vries
Thanks for perfectly reported issue.