Just to add info to this bug report, the traceback in the logs:
2019-04-16 07:16:09.070 20 INFO nova.compute.api [req-83e26ba8-f8ac-4b9b-9608-3c2aa7e9672d abaf4efd50ed493db1a49ce33d44717a 59ede4886038422f9d6c0cd324db4545 - default default] [instance: 3e7e8f96-9bb0-4d7f-b394-14bd3fb4ca17] Skipping quiescing instance: QEMU guest agent is not enabled
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
function_name, call_dict, binary, tb)
File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in exit
self.force_reraise()
File "/usr/lib/python3.6/site-packages/osloutils/excutils.py", line 196, in force_reraise
six.reraise(self.type, self.value, self.tb)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8491, in quiesce_instance
self.driver.quiesce(context, instance, image_meta)
File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2202, in quiesce
self._set_quiesced(context, instance, image_meta, True)
File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2181, in _set_quiesced
self._can_quiesce(instance, image_meta)
File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2175, in _can_quiesce
raise exception.QemuGuestAgentNotEnabled()
nova.exception.QemuGuestAgentNotEnabled: QEMU guest agent is not enabled
is not actually related to the job failure. The nova code will quiesce the server if it can before taking the volume snapshot, but if the qemu quest agent is not available, it will skip it and move on.
We should not be logging the traceback in nova, but it's happening because the exception is coming to nova-api remotely over RPC and the oslo.messaging code will automatically append the traceback to the exception object 'message' attribute, and we're currently logging 'LOG.info(..., exp)' there and thus inadvertently logging the traceback as well.
I'll propose a nova patch to stop logging the traceback, to help prevent future confusion.
Just to add info to this bug report, the traceback in the logs:
2019-04-16 07:16:09.070 20 INFO nova.compute.api [req-83e26ba8- f8ac-4b9b- 9608-3c2aa7e967 2d abaf4efd50ed493 db1a49ce33d4471 7a 59ede4886038422 f9d6c0cd324db45 45 - default default] [instance: 3e7e8f96- 9bb0-4d7f- b394-14bd3fb4ca 17] Skipping quiescing instance: QEMU guest agent is not enabled
Traceback (most recent call last):
File "/usr/lib/ python3. 6/site- packages/ oslo_messaging/ rpc/server. py", line 229, in inner
return func(*args, **kwargs)
File "/usr/lib/ python3. 6/site- packages/ nova/exception_ wrapper. py", line 79, in wrapped
function_name, call_dict, binary, tb)
File "/usr/lib/ python3. 6/site- packages/ oslo_utils/ excutils. py", line 220, in exit reraise( )
self.force_
File "/usr/lib/ python3. 6/site- packages/ osloutils/ excutils. py", line 196, in force_reraise self.type, self.value, self.tb)
six.reraise(
File "/usr/lib/ python3. 6/site- packages/ six.py" , line 693, in reraise
raise value
File "/usr/lib/ python3. 6/site- packages/ nova/exception_ wrapper. py", line 69, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/ python3. 6/site- packages/ nova/compute/ manager. py", line 8491, in quiesce_instance quiesce( context, instance, image_meta)
self.driver.
File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2202, in quiesce quiesced( context, instance, image_meta, True)
self._set_
File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2181, in _set_quiesced quiesce( instance, image_meta)
self._can_
File "/usr/lib/ python3. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2175, in _can_quiesce QemuGuestAgentN otEnabled( )
raise exception.
nova.exception. QemuGuestAgentN otEnabled: QEMU guest agent is not enabled
is not actually related to the job failure. The nova code will quiesce the server if it can before taking the volume snapshot, but if the qemu quest agent is not available, it will skip it and move on.
We should not be logging the traceback in nova, but it's happening because the exception is coming to nova-api remotely over RPC and the oslo.messaging code will automatically append the traceback to the exception object 'message' attribute, and we're currently logging 'LOG.info(..., exp)' there and thus inadvertently logging the traceback as well.
I'll propose a nova patch to stop logging the traceback, to help prevent future confusion.