OpenStack Compute (nova)

_HostMountState.mount() doesn't handle processutils.ProcessExecutionError properly

Bug #2032776 reported by Alexey Stupnikov on 2023-08-23

This bug report is a duplicate of: Bug #1984736: "TypeError: catching classes that do not inherit from BaseException is not allowed" is raised if volume mount fails in python3. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	In Progress	Medium	Alexey Stupnikov

Bug Description

The following error was caught while running downstream tempest resize test against stable/wallaby branch:

2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [req-2cb867f8-80e6-419d-9986-26309b832a92 44e407e8e5224e45b926d3b4924144fd 0f7fbf19b06245e7ad31f7e235fd312f - default default] [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Setting instance vm_state to ERROR: TypeError: catching classes that do not inherit from BaseException is not allowed
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Traceback (most recent call last):
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 305, in mount
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] nova.privsep.fs.mount(fstype, export, mountpoint, options)
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 253, in _wrap
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] return self.channel.remote_call(name, args, kwargs)
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 226, in remote_call
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] raise exc_type(*result[2])
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Command: mount -t nfs 192.168.1.50:/vol_cinder /var/lib/nova/mnt/724dab229d80c6a1a1e49a71c8356eed
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Exit code: 32
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Stdout: ''
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Stderr: 'Failed to connect to bus: No data available\nmount.nfs: access denied by server while mounting 192.168.1.50:/vol_cinder\n'
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90]
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] During handling of the above exception, another exception occurred:
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90]
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Traceback (most recent call last):
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10256, in _error_out_instance_on_exception
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] yield
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5851, in _finish_resize_helper
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] network_info = self._finish_resize(context, instance, migration,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5789, in _finish_resize
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] self._set_instance_info(instance, old_flavor)
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] self.force_reraise()
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] raise self.value
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5772, in _finish_resize
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] self.driver.finish_migration(context, migration, instance,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 11308, in finish_migration
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] xml = self._get_guest_xml(context, instance, network_info,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7182, in _get_guest_xml
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] conf = self._get_guest_config(instance, network_info, image_meta,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 6789, in _get_guest_config
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] storage_configs = self._get_guest_storage_config(context,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5395, in _get_guest_storage_config
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] self._connect_volume(context, connection_info, instance)
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 1883, in _connect_volume
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] vol_driver.connect_volume(connection_info, instance)
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/fs.py", line 113, in connect_volume
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] mount.mount(self.fstype, export, vol_name, mountpoint, instance,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 407, in mount
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] mount_state.mount(fstype, export, vol_name, mountpoint, instance,
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/volume/mount.py", line 306, in mount
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] except processutils.ProcessExecutionError():
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] TypeError: catching classes that do not inherit from BaseException is not allowed
2023-08-15 03:14:03.764 2 ERROR nova.compute.manager [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90]
2023-08-15 03:14:03.872 2 INFO nova.compute.manager [req-2cb867f8-80e6-419d-9986-26309b832a92 44e407e8e5224e45b926d3b4924144fd 0f7fbf19b06245e7ad31f7e235fd312f - default default] [instance: 70e4550d-c03c-4f90-bdb2-fb6e082cfc90] Deleting allocations for old flavor on source node compute-1.redhat.local after finish_resize failure. You may be able to recover the instance by hard rebooting it.

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=2232232

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-23: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/892496

Changed in nova:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-23: Change abandoned on nova (master)

Change abandoned by "Alexey Stupnikov <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/892496
Reason: This is already addressed in I24a269f9809063f864a4f0443d6724dfa8703d9d

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1984736 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.