check_instance_shared_storage RPC call is broken

Bug #1921399 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith

Bug Description

We broke check_instance_shared_storage() in this change:

https://review.opendev.org/c/openstack/nova/+/761452/13..15/nova/compute/rpcapi.py

Where we re-ordered the rpcapi client signature without adjusting the caller. This leads to this failure:

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] Traceback (most recent call last):

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/new/nova/nova/compute/manager.py", line 797, in _is_instance_storage_shared

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] instance, data, host=host))

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 618, in check_instance_shared_storage

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return cctxt.call(ctxt, 'check_instance_shared_storage', **msg_args)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 179, in call

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] transport_options=self.transport_options)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] transport_options=transport_options)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] transport_options=transport_options)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] raise result

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] AttributeError: 'Instance' object has no attribute 'filename'

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] Traceback (most recent call last):

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] res = self.dispatcher.dispatch(message)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return self._do_dispatch(endpoint, method, ctxt, args)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] result = func(ctxt, **new_args)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/old/nova/nova/exception_wrapper.py", line 78, in wrapped

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] function_name, call_dict, binary)

Mar 25 13:46:28.041587 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] self.force_reraise()

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] six.reraise(self.type_, self.value, self.tb)

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] raise value

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/old/nova/nova/exception_wrapper.py", line 69, in wrapped

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return f(self, context, *args, **kw)

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/old/nova/nova/compute/manager.py", line 7792, in check_instance_shared_storage

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return self.driver.check_instance_shared_storage_remote(ctxt, data)

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 8550, in check_instance_shared_storage_remote

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return os.path.exists(data['filename'])

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 755, in __getitem__

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] return getattr(self, name)

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef] AttributeError: 'Instance' object has no attribute 'filename'

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Mar 25 13:46:28.044171 ubuntu-bionic-vexxhost-ca-ymq-1-0023683006 nova-compute[8570]: ERROR nova.compute.manager [instance: 20d48d76-f93c-4b3c-90a8-cd7f654b28ef]

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :
Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

Fix merged to master, and will be part of Wallaby RC1.

Changed in nova:
status: In Progress → Fix Released
milestone: xena-rc1 → wallaby-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.0.0.0rc1

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.