Live Migration Failures

[Paul Bessa]

Live Migration of Cinder Fiber Channel backend devices are failing with the following error message in the nova-compute logs.

 '/usr/bin/nova-rootwrap: Executable not found: sg_scan (filter match = sg_scan)\n'

Steps to reproduce:

 1) Deploy FC backend array enabled for Cinder block storage.
 2) Create a volume from an image.
 3) Boot an instance from that volume.
 4) Run I/O workload
 4) Live migrate that instance.
 Expected result: live migration successful.

fuel --fuel-version
api: '1'
auth_required: true
feature_groups: []
openstack_version: mitaka-9.0
release: '9.2'

fuel node list
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+--------+-----------------+---------+-----------+-------------------+--------------------+---------------+--------+---------
 7 | ready | LHXSER3 (26:04) | 2 | 10.20.0.4 | a0:36:9f:5a:26:04 | cinder, controller | | 1 | 2
 6 | ready | LHXSER2 (26:ac) | 2 | 10.20.0.3 | a0:36:9f:5a:26:ac | compute | | 1 | 2
 5 | ready | LHXSER4 (26:20) | 2 | 10.20.0.5 | a0:36:9f:5a:26:20 | compute | | 1 | 2

Fuel diagnotic snapshot is available if needed.
Cinder-volume and nova-compute log files are attachd.

[Paul Bessa (paul-bessa)]

Nova Compute log.

[Oleksiy Molchanov (omolchanov)]

Hi Paul,

How did you enable this backend?

[Paul Bessa (paul-bessa)]

I have a multiple backed configuration, below is an excerpt of the enabled backend configuration in the "default" section of the cinder.conf file. The live migration operation is failing/hanging indefinitely for the Cinder based backends.

Instance Live Migration is successful if only the LVM-backend is enabled.

enabled_backends = vnx_225_fc, xio_039_fc, vmax_1176_fc

[vnx_225_fc]

volume_backend_name = vnx_225_fc
storage_vnx_pool_names = OSPQ
san_thin_provision = true
san_ip = 10.228.227.225
san_secondary_ip = 10.228.227.226
#VNX user name
san_login = sysadmin
#VNX user password
san_password = sysadmin
#VNX user type. Valid values are: global(default), local and ldap.
storage_vnx_authentication_type = global #Directory path of the VNX security file. Make sure the security file is generated first.
#VNX credentials are not necessary when using security file.
#storage_vnx_security_file_dir = /etc/secfile/array1 naviseccli_path = /opt/Navisphere/bin/naviseccli #timeout in minutes default_timeout = 10 #If deploying EMCCLIISCSIDriver:
#volume_driver=cinder.volume.drivers.emc.emc_cli_iscsi.EMCCLIISCSIDriver
volume_driver=cinder.volume.drivers.emc.emc_cli_fc.EMCCLIFCDriver
destroy_empty_storage_group = False
initiator_auto_registration = True
num_volume_device_scan_tries = 10
use_multipath_for_image_xfer = True
enforce_multipath_for_image_xfer = True

[xio_039_fc]

volume_backend_name = xio_039_fc
volume_driver=cinder.volume.drivers.emc.xtremio.XtremIOFibreChannelDriver
san_ip = 10.108.225.123
xtremio_cluster_name = 96Y-XIO
san_login = admin
san_password = Xtrem10
max_over_subscription_ratio = 40.0
use_multipath_for_image_xfer = True
enforce_multipath_for_image_xfer = True
num_volume_device_scan_tries = 10

[vmax_1176_fc]

volume_backend_name = vmax_1176_fc
volume_driver = cinder.volume.drivers.emc.emc_vmax_fc.EMCVMAXFCDriver
cinder_emc_config_file = /etc/cinder/cinder_emc_config_vmax_1176_fc.xml
use_multipath_for_image_xfer = True
enforce_multipath_for_image_xfer = True
num_volume_device_scan_tries = 10