OpenStack Compute (nova)

Nova-compute doesn't mount volume filesystems on resume

Bug #1845199 reported by Jacek Tomasiak
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Jacek Tomasiak

Bug Description

Description
===========
Suspended instances with volumes attached fail to resume if the /var/lib/nova/mnt/... filesystems are not mounted. The mounts could be lost due to e.g. node reboot during upgrade.

Steps to reproduce
==================
* Boot some instance and attach volume stored in NFS (/var/lib/nova/mnt/... should get mounted on the compute node)
* Suspend the instance ($ nova suspend <instance id or name>)
* Unmount the /var/lib/nova/mnt/... filesystem
* Try to resume the instance ($ nova resume <instance id or name>)

Expected result
===============
The instance should become ACTIVE.

Actual result
=============
Instance switches to ERROR state.

Environment
===========
openstack-nova-compute-18.2.3~dev10-3.5.noarch
python-novaclient-11.0.0-3.126.noarch
python-nova-18.2.3~dev10-3.5.noarch
openstack-nova-18.2.3~dev10-3.5.noarch

Hypervisor: Libvirt + KVM

Storage: We experienced this with NFS Cinder backend but probably other filesystem-based volume types are affected as well.

Network: Neutron

Logs & Configs
==============
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [req-38561c2e-7312-47c8-aec9-40bbe24a5b8a 50269c9569ba4c388a96dd8eaa7106b2 3dab9b31bfaf4298801706aaf14ce83a - default default] [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Setting instance vm_state to ERROR: libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] Traceback (most recent call last):
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8208, in _error_out_instance_on_exception
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] yield
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5027, in resume_instance
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] block_device_info)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2997, in resume
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] vifs_already_plugged=True)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5713, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] destroy_disks_on_failure)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5682, in _create_domain_and_network
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] post_xml_callback=post_xml_callback)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5616, in _create_domain
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] guest.launch(pause=pause)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self._encoded_xml, errors='ignore')
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] self.force_reraise()
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(self.type_, self.value, self.tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] return self._domain.createWithFlags(flags)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] result = proxy_call(self._autowrap, f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = execute(f, *args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] six.reraise(c, e, tb)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] rv = meth(*args, **kwargs)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1099, in createWithFlags
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2019-09-20 08:01:00.619 24744 ERROR nova.compute.manager [instance: a7cf0dc4-732c-4d82-97c7-9a191b347a20] libvirtError: Cannot access storage file '/var/lib/nova/mnt/3cc56405338b71a0b5bad6102bf7299b/volume-b8bc9d61-1f2f-461d-a787-7b912e8cc8fa': No such file or directory

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/684290

Changed in nova:
assignee: nobody → Jacek Tomasiak (skazi)
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.