OpenStack Compute (nova)

nova fails to rescue an instance because ramdisk file doesn't exist

Bug #1687086 reported by Ihar Hrachyshka on 2017-04-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Expired	Undecided	Unassigned

Bug Description

http://logs.openstack.org/67/457467/4/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/4d6be0a/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-04-20_16_18_59_065

2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [req-26543eff-dd70-4526-bec6-fc977ea734dc tempest-ServerRescueNegativeTestJSON-295821689 tempest-ServerRescueNegativeTestJSON-295821689] [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] Error trying to Rescue Instance
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] Traceback (most recent call last):
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/opt/stack/new/nova/nova/compute/manager.py", line 3370, in rescue_instance
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] rescue_image_meta, admin_password)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2636, in rescue
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] self._create_domain(xml, post_xml_callback=gen_confdrive)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5002, in _create_domain
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] guest.launch(pause=pause)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 145, in launch
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] self._encoded_xml, errors='ignore')
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] self.force_reraise()
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] six.reraise(self.type_, self.value, self.tb)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 140, in launch
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] return self._domain.createWithFlags(flags)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] rv = execute(f, *args, **kwargs)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] six.reraise(c, e, tb)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] rv = meth(*args, **kwargs)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1065, in createWithFlags
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb] libvirtError: unable to stat: /opt/stack/data/nova/instances/6e63ccaa-f174-4371-a169-d5303db821eb/ramdisk.rescue: No such file or directory
2017-04-20 16:18:59.065 2166 ERROR nova.compute.manager [instance: 6e63ccaa-f174-4371-a169-d5303db821eb]
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server [req-26543eff-dd70-4526-bec6-fc977ea734dc tempest-ServerRescueNegativeTestJSON-295821689 tempest-ServerRescueNegativeTestJSON-295821689] Exception during message handling
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 75, in wrapped
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 66, in wrapped
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 188, in decorated_function
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 157, in decorated_function
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/utils.py", line 686, in decorated_function
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 216, in decorated_function
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 204, in decorated_function
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3377, in rescue_instance
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server reason=_("Driver Error: %s") % e)
2017-04-20 16:18:59.523 2166 ERROR oslo_messaging.rpc.server InstanceNotRescuable: Instance 6e63ccaa-f174-4371-a169-d5303db821eb cannot be rescued: Driver Error: unable to stat: /opt/stack/data/nova/instances/6e63ccaa-f174-4371-a169-d5303db821eb/ramdisk.rescue: No such file or directory

This seems to happen for multiple instances in the same test run. Nothing suspicious that I could spot in libvirt logs.

It's Ocata but I am not sure it's not present in master.

Tags:

Ihar Hrachyshka (ihar-hrachyshka) on 2017-04-28

tags:

added: gate-failure

Revision history for this message

Sean Dague (sdague) wrote on 2017-06-27:

Automatically discovered version ocata in description. If this is incorrect, please update the description to include 'nova version: ...'

tags:

added: openstack-version.ocata

Revision history for this message

Sean Dague (sdague) wrote on 2017-07-27:

This is linking to a gate run that no longer has logs. Set to incomplete. Will expire out if there are no updated logs.

Changed in nova:
status:	New → Incomplete

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-09-26:

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.