reboot test fails in gate-grenade-dsvm-multinode with missing disk path

[Matt Riedemann]

http://logs.openstack.org/44/245344/4/gate/gate-grenade-dsvm-multinode/1840523/logs/subnode-2/old/screen-n-cpu.txt.gz?level=TRACE#_2015-12-01_23_58_04_999

2015-12-01 23:58:04.999 ERROR nova.compute.manager [req-8e76a5ba-f89f-4bfe-8218-aeaa580a6e13 tempest-ServerActionsTestJSON-475123373 tempest-ServerActionsTestJSON-884217194] [instance: e50d2ceb-2be4-4c1a-a190-4aa9ab160af9] Cannot reboot instance: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/e50d2ceb-2be4-4c1a-a190-4aa9ab160af9/disk'
2015-12-01 23:58:05.513 ERROR oslo_messaging.rpc.dispatcher [req-8e76a5ba-f89f-4bfe-8218-aeaa580a6e13 tempest-ServerActionsTestJSON-475123373 tempest-ServerActionsTestJSON-884217194] Exception during message handling: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/e50d2ceb-2be4-4c1a-a190-4aa9ab160af9/disk'
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/exception.py", line 89, in wrapped
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher payload)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/exception.py", line 72, in wrapped
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 350, in decorated_function
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 323, in decorated_function
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 400, in decorated_function
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 378, in decorated_function
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 366, in decorated_function
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 2954, in reboot_instance
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher self._set_instance_obj_error_state(context, instance)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/compute/manager.py", line 2935, in reboot_instance
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher bad_volumes_callback=bad_volumes_callback)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2055, in reboot
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher block_device_info)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2152, in _hard_reboot
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher block_device_info)
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6484, in _get_instance_disk_info
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher dk_size = int(os.path.getsize(path))
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/genericpath.py", line 49, in getsize
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher return os.stat(filename).st_size
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher OSError: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/e50d2ceb-2be4-4c1a-a190-4aa9ab160af9/disk'
2015-12-01 23:58:05.513 31663 ERROR oslo_messaging.rpc.dispatcher

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22Cannot%20reboot%20instance:%20%5BErrno%202%5D%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22

122 hits in 7 days, check and gate, master and stable/liberty, all failures.

[Matt Riedemann (mriedem)]

Looks like the problem, at least in this case:

http://logs.openstack.org/32/253732/4/check/gate-grenade-dsvm-multinode/9f726ed/logs/

Is that the two nodes have the same hostname (ubuntu in this case), which is confusing things.

[Matt Riedemann (mriedem)]

BTW, credit to dansmith for noticing the hostname problem in comment 1.

[Matt Riedemann (mriedem)]

This infra patch should help set the hostname: https://review.openstack.org/#/c/254858/

[Matt Riedemann (mriedem)]

Fixed by infra.