libvirt ERROR in n-cpu log after successful tempest run

[David Kranz]

This is happening a lot. Example: http://logs.openstack.org/58/53258/1/check/check-tempest-devstack-vm-full/77c44f5/logs/screen-n-cpu.txt.gz

2013-10-23 02:15:42.433 27883 INFO nova.virt.libvirt.driver [-] [instance: f2b25a0f-5347-473f-a3b3-ff27d9b558e4] Instance destroyed successfully.
2013-10-23 02:15:42.435 ERROR nova.virt.libvirt.driver [req-1d3ac496-38b6-4e07-8bd3-8a006d6722e0 None None] [instance: afc357da-8ee6-4a01-a40d-81468c1e1a5a] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'afc357da-8ee6-4a01-a40d-81468c1e1a5a'

[Kashyap Chamarthy (kashyapc)]

This is what it seems to be hitting:

    https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L809

I think it's trying to "destroy" (i.e. "force stop" in libvirt's parlance) a non-existent domain -- which is not really catastrophic, and that's why Tempest runs must be succeeding?

[Kashyap Chamarthy (kashyapc)]

[Going through this bug as part of periodic triage.]

I just checked for the past few weeks successful tempest runs, I can't find the error message noted in the Description.

    http://logs.openstack.org/03/67703/3/check/check-tempest-dsvm-postgres-full/de93545/logs/screen-n-cpu.txt.gz

While I agree that it'd be nice to pin point the root cause of the issue, but I don't think it's worth the time to investigate now as it's not occurring consistently.

Please re-open this bug if it occurs again.

[Kashyap Chamarthy (kashyapc)]

Sorry, I was wrong.

From logstash, it seems to happen consistently -- http://goo.gl/w2wqP1

---------------------
[. . .]
2014-03-03T09:03:45.000 [req-40bc74a9-3232-4398-82f4-33d7c0bfbf3b None None] [instance: a6477b06-e52c-4896-929a-81d858ed595d] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'a6477b06-e52c-4896-929a-81d858ed595d'
2014-03-03T07:37:58.000 [req-bbd7f1b8-78b9-47f2-8caa-162c869d250d None None] [instance: 8e34bcb4-6f64-4a92-b3db-7756f4473c09] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '8e34bcb4-6f64-4a92-b3db-7756f4473c09'
2014-03-03T06:26:30.000 [req-6ceae915-68eb-4b2d-b1db-34cde77be7b6 None None] [instance: 418e2339-7ec5-4180-b70b-d3da2c376943] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '418e2339-7ec5-4180-b70b-d3da2c376943'
2014-03-03T06:24:35.000 [req-e09878a1-6115-45b7-b93b-04c738b21ea2 None None] [instance: f2c79fa8-bf0b-425f-bf17-43bb8f38cb23] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'f2c79fa8-bf0b-425f-bf17-43bb8f38cb23'
2014-03-03T06:24:31.000 [req-eec56a5a-887f-4181-9aff-98b98cc827a2 None None] [instance: 02ebc93a-5721-4a24-87b1-e4d75b6ab0c5] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '02ebc93a-5721-4a24-87b1-e4d75b6ab0c5'
2014-03-03T04:58:28.000 2014-03-03 04:22:42.689 ERROR nova.virt.libvirt.driver [req-631f31ff-644f-423e-9875-5af1373f1e0f AttachInterfacesTestJSON-920406155 AttachInterfacesTestJSON-799810151] [instance: fa3921d4-8f5a-4228-945f-1edd390cefb4] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T04:44:09.000 [req-0a0edf29-9f1f-4c1f-9184-3ab4fa8ef961 None None] [instance: bd6367f1-4d2b-436a-a323-ae4ffdeb98a7] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'bd6367f1-4d2b-436a-a323-ae4ffdeb98a7'
2014-03-03T04:22:42.000 [req-631f31ff-644f-423e-9875-5af1373f1e0f AttachInterfacesTestJSON-920406155 AttachInterfacesTestJSON-799810151] [instance: fa3921d4-8f5a-4228-945f-1edd390cefb4] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T04:13:59.000 [req-7b0f5540-eb44-44a5-b9b1-37d08884a6b0 None None] [instance: bec03c58-ceb4-4e19-8492-779ac61450c0] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'bec03c58-ceb4-4e19-8492-779ac61450c0'
2014-03-03T01:38:22.000 2014-03-03 01:25:22.639 ERROR nova.virt.libvirt.driver [req-2ba196b9-c2be-4f27-a5bf-3c19f6bb33a4 AttachInterfacesV3Test-58929740 AttachInterfacesV3Test-599922619] [instance: 1f5a6fc3-a8a7-4183-8c19-0db0cd450d80] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T01:25:22.000 [req-2ba196b9-c2be-4f27-a5bf-3c19f6bb33a4 AttachInterfacesV3Test-58929740 AttachInterfacesV3Test-599922619] [instance: 1f5a6fc3-a8a7-4183-8c19-0db0cd450d80] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-...

[Kashyap Chamarthy (kashyapc)]

I had a brief chat with Daniel Berrangé on #openstack-nova, and like I noted in comment #1, he has this to add:

    [instance: 8e34bcb4-6f64-4a92-b3db-7756f4473c09] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '8e34bcb4-6f64-4a92-b3db-7756f4473c09'

means:

Nova is trying to tell libvirt to destroy a guest which has already gone. As an error, it is pretty harmless from libvirt's POV, but it might indicate an unexpected scenario from Nova's POV.

[Joe Gordon (jogo)]

See https://blueprints.launchpad.net/nova/+spec/support-libvirt-1x