OpenStack Compute (nova)

Extend attached volume case failed

Bug #1849425 reported by hjy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned
OpenStack Compute (nova)
Fix Released
Undecided
Unassigned

Bug Description

The online expansion volume use case failed, tempest.api.volume.test_volumes_extend:VolumesExtendAttachedTest.test_extend_attached_volume. This use case was originally successfully executed and failed after reinstalling devstack (stein-->train). The cinder driver has not changed, and the multipath.conf configuration has not changed. There are two kinds of errors in <email address hidden>, one is the error under the ubuntu18 system, and the other is the error under the centos7 system. as follows

ubuntu18:
Oct 23 09:34:52 stack nova-compute[11706]: WARNING nova.compute.manager [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] [instance: 6a4b5b05-20b7-46c1-b903-b6fe40a38aad] Extend volume failed, volume_id=bc3c5af6-b6c6-4e01-8d0e-774ad637debc, reason: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server [req-0ce8f571-5c04-4a72-8049-3839c934c7c9 req-421b45b2-be78-4bc3-8a1b-c469e4d7504d service nova] Exception during message handling: TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/dist-packages/six.py", line 693, in reraise
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server raise value
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server bdm.volume_size * units.Gi)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1768, in extend_volume
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server dev.resize(new_size // units.Ki)
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for //: 'NoneType' and 'int'
Oct 23 09:34:52 stack nova-compute[11706]: ERROR oslo_messaging.rpc.server

centos7:
Oct 10 10:15:12 devstack nova-compute[59879]: DEBUG os_brick.initiator.connectors.iscsi [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] <== extend_volume: exception (371ms) ProcessExecutionError('fail\n', '', 1, 'multipathd resize map 3600b34245364482d2d43d02ffda900df', None) {{(pid=59879) trace_logging_wrapper /usr/local/lib/python3.7/site-packages/os_brick/utils.py:156}}
Oct 10 10:15:12 devstack nova-compute[59879]: WARNING nova.compute.manager [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] [instance: 72d1a0fc-91bf-46eb-9ef8-96a81886a747] Extend volume failed, volume_id=79481f44-72dd-4c07-98cb-70f52048f10e, reason: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server [req-fbe8d107-47e2-44b5-af76-413b6e6a24ed req-af3e94f4-cb50-4f92-90c1-64eaf74727b1 service nova] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8587, in external_instance_event
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.extend_volume(context, instance, event.tag)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1346, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 215, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server self.force_reraise()
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise value
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 8545, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server bdm.volume_size * units.Gi)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1738, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1371, in _extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server requested_size)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 86, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server new_size = self.connector.extend_volume(connection_info['data'])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 477, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self._linuxscsi.extend_volume(volume_paths)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 572, in extend_volume
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = self.multipath_resize_map(scsi_wwn)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/initiator/linuxscsi.py", line 531, in multipath_resize_map
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server root_helper=self._root_helper)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/executor.py", line 52, in _execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server result = self.__execute(*args, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return execute_root(*cmd, **kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs)
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Command: multipathd resize map 3600b34245364482d2d43d02ffda900df
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Exit code: 1
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stdout: 'fail\n'
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server Stderr: ''
Oct 10 10:15:12 devstack nova-compute[59879]: ERROR oslo_messaging.rpc.server

These two errors will be resolved after changing user_friendly_names in multipath.conf to no. What is the problem with this?

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

First traceback appears to initiate from the Nova libvirt driver.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Hmm, https://review.opendev.org/#/c/613039/ touched that code in Train to add extend volume support for the rbd volume driver. What is the volume type that you're using when you hit this? I'm wondering if the _extend_volume method in the libvirt driver is returning None.

Changed in nova:
status: New → Incomplete
tags: added: libvirt volumes
Revision history for this message
hjy (ruffian-sheep) wrote :

I did not set the volume type, just executed the tempest test case.

Revision history for this message
Lee Yarwood (lyarwood) wrote :

> These two errors will be resolved after changing user_friendly_names
> in multipath.conf to no. What is the problem with this?

IIRC os-brick had issues identifying the correct underlying mpath device with friendly names was enabled prior to this bug being opened and that likely caused the failure to resize the device on CentOS and caused os-brick to return new_size as None on Ubuntu causing the TypeError.

For the latter n-cpu can catch and raise this cleanly before we ever get to resizing the attached device, I'll push a change for that now but close this out otherwise for cinder and os-brick. If you're somehow still having issues then please raise a bug directly against os-brick for the map resize issues.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/801714

Changed in nova:
status: Incomplete → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/801714
Committed: https://opendev.org/openstack/nova/commit/49ba5a763f9d2c7c496f391ca8149a18541bfec7
Submitter: "Zuul (22348)"
Branch: master

commit 49ba5a763f9d2c7c496f391ca8149a18541bfec7
Author: Lee Yarwood <email address hidden>
Date: Thu Jul 22 09:05:51 2021 +0100

    libvirt: Handle silent failures to extend volume within os-brick

    As seen in bug #1849425 os-brick can fail to extend an underlying volume
    device on the compute silently, returning a new_size of None to n-cpu.

    While this should ultimatley be addressed in os-brick n-cpu can also
    handle this before we eventually run into a type error when attemting
    floor division later in the volume extend flow.

    Change-Id: Ic8091537274a5ad27fb5af8939f81ed154b7ad7c
    Closes-Bug: #1849425

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.