os-brick

[Netapp Ontap] iSCSI multipath flush fail with stderr: map in use

  1. Series victoria
  2. Bug #1992289
Bug #1992289 reported by Sang Tran
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
os-brick
New
Undecided
Unassigned
Victoria
New
Undecided
Unassigned

Bug Description

This error happens when I live-migrate or resize an instance to another compute host. I also checked the SAN configuration to ensure the corresponding LUN was completely changed to a new host's initiator and LUN-ID. My Openstack version is Victoria and Netapp Ontap driver for SAN.

After this error, the instance is marked ERROR and needs to reset state then manually do the command "dmsetup remove -f <mapper-id>" before starting VM.

2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server [req-166755c4-f6b9-48e8-a607-25c4b4787f45 183ef8f0f8244784b6805cb0ca8613e8 c881b9aed5964cc2973bd6a68a4191c6 - default default] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -f 3600a09803831486e695d536269665144
Exit code: 1
Stdout: ''
Stderr: 'Oct 04 10:49:32 | 3600a09803831486e695d536269665144: map in use\n'
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server _emit_exception_notification(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9155, in rollback_live_migration_at_destination
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.driver.rollback_live_migration_at_destination(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9677, in rollback_live_migration_at_destination
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.destroy(context, instance, network_info, block_device_info,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1342, in destroy
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.cleanup(context, instance, network_info, block_device_info,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1410, in cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._cleanup(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1471, in _cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server LOG.warning(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise value
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1463, in _cleanup
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self._disconnect_volume(context, connection_info, instance)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 1735, in _disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server vol_driver.disconnect_volume(connection_info, instance)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.connector.disconnect_volume(connection_info['data'], None)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/utils.py", line 144, in trace_logging_wrapper
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 877, in disconnect_volume
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self._cleanup_connection(connection_properties, force=force,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/connectors/iscsi.py", line 927, in _cleanup_connection
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server multipath_name = self._linuxscsi.remove_connection(
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 313, in remove_connection
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self.flush_multipath_device(multipath_name)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/initiator/linuxscsi.py", line 369, in flush_multipath_device
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server self._execute('multipath', '-f', device_map_name, run_as_root=True,
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/executor.py", line 52, in _execute
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Command: multipath -f 3600a09803831486e695d536269665144
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Exit code: 1
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Stdout: ''
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server Stderr: 'Oct 04 10:49:32 | 3600a09803831486e695d536269665144: map in use\n'
2022-10-04 10:50:12.238 7 ERROR oslo_messaging.rpc.server

Tags: drivers netapp
Sofia Enriquez (lsofia-enriquez)
tags: added: drivers netapp
summary: - iSCSI multipath flush fail with stderr: map in use
+ [Netapp Ontap] iSCSI multipath flush fail with stderr: map in use
Revision history for this message
Khoi (khoinh5) wrote :

Hello.

I have same problem here. It happens sometime when I resize instance or live migrate. I hope we can some solutions for this.

I use Yoga and Dell Powerstore 5000T ISCSI.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.