Nova-compute doesn't mount volume filesystems on resume

Bug #1845199 reported by Jacek Tomasiak on 2019-09-24
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Jacek Tomasiak

Bug Description

Description
===========
Suspended instances with volumes attached fail to resume if the /var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due to e.g. node reboot during upgrade.

Steps to reproduce
==================
* Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/... should get mounted on the compute node)
* Suspend the instance ($ nova suspend <instance id or name>)
* Unmount the /var/lib/nova/mnt/... filesystem
* Try to resume the instance ($ nova resume <instance id or name>)

Expected result
===============
The instance should become ACTIVE.

Actual result
=============
Instance switches to ERROR state.

Environment
===========
openstack-nova-compute-18.2.3~dev10-3.5.noarch
python-novaclient-11.0.0-3.126.noarch
python-nova-18.2.3~dev10-3.5.noarch
openstack-nova-18.2.3~dev10-3.5.noarch

Hypervisor: Libvirt + KVM

Storage: We experienced this with NFS Cinder backend but probably other filesystem-based volume types are affected as well.

Network: Neutron

Logs & Configs
==============
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2 3dab9b31bfaf4298801706aaf14ce83a - default default] [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR: libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in _error_out_instance_on_exception
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] yield
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in resume_instance
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] block_device_info)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in resume
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] vifs_already_plugged=True)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] destroy_disks_on_failure)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] post_xml_callback=post_xml_callback)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in _create_domain
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] guest.launch(pause=pause)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self._encoded_xml, errors='ignore')
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] return self._domain.createWithFlags(flags)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = execute(f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(c, e, tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = meth(*args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory

Fix proposed to branch: master
Review: https://review.opendev.org/684290

Changed in nova:
assignee: nobody → Jacek Tomasiak (skazi)
status: New → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers