IOError: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/UUID/libvirt.xml'

[Joe Gordon]

2014-02-27 22:17:30.887 26111 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/32ce67c4-8444-4f3a-8fda-58c8e303efdc/libvirt.xml'
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 88, in wrapped
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher payload)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 71, in wrapped
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 243, in decorated_function
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher pass
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 229, in decorated_function
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 294, in decorated_function
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 271, in decorated_function
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 258, in decorated_function
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 2076, in start_instance
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher self._power_on(context, instance)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 2064, in _power_on
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2093, in power_on
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher self._hard_reboot(context, instance, network_info, block_device_info)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2045, in _hard_reboot
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher write_to_disk=True)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3363, in to_xml
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher libvirt_utils.write_to_file(xml_path, xml)
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/utils.py", line 526, in write_to_file
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher with open(path, 'w') as f:
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/opt/stack/data/nova/instances/32ce67c4-8444-4f3a-8fda-58c8e303efdc/libvirt.xml'
2014-02-27 22:17:30.887 26111 TRACE oslo.messaging.rpc.dispatcher

logstash query: message:"No such file or directory" AND message:"libvirt.xml" AND filename:"logs/screen-n-cpu.txt" AND message:"Exception during message handling"

[Joe Gordon (jogo)]

http://logs.openstack.org/55/76755/2/check/check-tempest-dsvm-postgres-full/4282682/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-02-27_22_17_30_887

[Matt Riedemann (mriedem)]

related to bug 1277027 or patch https://review.openstack.org/#/c/71709/ ?

[Matt Riedemann (mriedem)]

Maybe related to bug 1260644 or bug 1255624?

[Matt Riedemann (mriedem)]

5 hits in the last 7 days, looks like it started showing up around 2/24:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeVwiIEFORCBtZXNzYWdlOlwibGlidmlydC54bWxcIiBBTkQgZmlsZW5hbWU6bG9ncypzY3JlZW4tbi1jcHUudHh0IEFORCBtZXNzYWdlOlwiRXhjZXB0aW9uIGR1cmluZyBtZXNzYWdlIGhhbmRsaW5nXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTM1NTU0OTc0NDV9

[Matt Riedemann (mriedem)]

If you expand the query it showed up as early as 2/18 actually:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeVwiIEFORCBtZXNzYWdlOlwibGlidmlydC54bWxcIiBBTkQgZmlsZW5hbWU6bG9ncypzY3JlZW4tbi1jcHUudHh0IEFORCBtZXNzYWdlOlwiRXhjZXB0aW9uIGR1cmluZyBtZXNzYWdlIGhhbmRsaW5nXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0wMi0xNFQwMjo0NDo1MyswMDowMCIsInRvIjoiMjAxNC0wMi0yOFQwMjo0NDo1MyswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxMzkzNTU1NzI5MzYzfQ==

[Jeremy Stanley (fungi)]

The current query in e-r for this is matching more successful runs than failures, so is probably not a good indicator.

[Dan Genin (daniel-genin)]

Not getting any hits at all in logstash. Has this been inadvertently fixed?

[Joe Gordon (jogo)]

I don't see any hits either looks like this may have been fixed.