libvirt ERROR in n-cpu log after successful tempest run

Bug #1243823 reported by David Kranz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Low
Unassigned

Bug Description

This is happening a lot. Example: http://logs.openstack.org/58/53258/1/check/check-tempest-devstack-vm-full/77c44f5/logs/screen-n-cpu.txt.gz

2013-10-23 02:15:42.433 27883 INFO nova.virt.libvirt.driver [-] [instance: f2b25a0f-5347-473f-a3b3-ff27d9b558e4] Instance destroyed successfully.
2013-10-23 02:15:42.435 ERROR nova.virt.libvirt.driver [req-1d3ac496-38b6-4e07-8bd3-8a006d6722e0 None None] [instance: afc357da-8ee6-4a01-a40d-81468c1e1a5a] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'afc357da-8ee6-4a01-a40d-81468c1e1a5a'

Tags: libvirt
melanie witt (melwitt)
tags: added: libvirt
summary: - libvert ERROR in n-cpu log after successful tempest run
+ libvirt ERROR in n-cpu log after successful tempest run
Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :

This is what it seems to be hitting:

    https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L809

I think it's trying to "destroy" (i.e. "force stop" in libvirt's parlance) a non-existent domain -- which is not really catastrophic, and that's why Tempest runs must be succeeding?

Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :

[Going through this bug as part of periodic triage.]

I just checked for the past few weeks successful tempest runs, I can't find the error message noted in the Description.

    http://logs.openstack.org/03/67703/3/check/check-tempest-dsvm-postgres-full/de93545/logs/screen-n-cpu.txt.gz

While I agree that it'd be nice to pin point the root cause of the issue, but I don't think it's worth the time to investigate now as it's not occurring consistently.

Please re-open this bug if it occurs again.

Changed in nova:
status: New → Incomplete
Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :
Download full text (4.4 KiB)

Sorry, I was wrong.

From logstash, it seems to happen consistently -- http://goo.gl/w2wqP1

---------------------
[. . .]
2014-03-03T09:03:45.000 [req-40bc74a9-3232-4398-82f4-33d7c0bfbf3b None None] [instance: a6477b06-e52c-4896-929a-81d858ed595d] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'a6477b06-e52c-4896-929a-81d858ed595d'
2014-03-03T07:37:58.000 [req-bbd7f1b8-78b9-47f2-8caa-162c869d250d None None] [instance: 8e34bcb4-6f64-4a92-b3db-7756f4473c09] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '8e34bcb4-6f64-4a92-b3db-7756f4473c09'
2014-03-03T06:26:30.000 [req-6ceae915-68eb-4b2d-b1db-34cde77be7b6 None None] [instance: 418e2339-7ec5-4180-b70b-d3da2c376943] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '418e2339-7ec5-4180-b70b-d3da2c376943'
2014-03-03T06:24:35.000 [req-e09878a1-6115-45b7-b93b-04c738b21ea2 None None] [instance: f2c79fa8-bf0b-425f-bf17-43bb8f38cb23] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'f2c79fa8-bf0b-425f-bf17-43bb8f38cb23'
2014-03-03T06:24:31.000 [req-eec56a5a-887f-4181-9aff-98b98cc827a2 None None] [instance: 02ebc93a-5721-4a24-87b1-e4d75b6ab0c5] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '02ebc93a-5721-4a24-87b1-e4d75b6ab0c5'
2014-03-03T04:58:28.000 2014-03-03 04:22:42.689 ERROR nova.virt.libvirt.driver [req-631f31ff-644f-423e-9875-5af1373f1e0f AttachInterfacesTestJSON-920406155 AttachInterfacesTestJSON-799810151] [instance: fa3921d4-8f5a-4228-945f-1edd390cefb4] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T04:44:09.000 [req-0a0edf29-9f1f-4c1f-9184-3ab4fa8ef961 None None] [instance: bd6367f1-4d2b-436a-a323-ae4ffdeb98a7] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'bd6367f1-4d2b-436a-a323-ae4ffdeb98a7'
2014-03-03T04:22:42.000 [req-631f31ff-644f-423e-9875-5af1373f1e0f AttachInterfacesTestJSON-920406155 AttachInterfacesTestJSON-799810151] [instance: fa3921d4-8f5a-4228-945f-1edd390cefb4] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T04:13:59.000 [req-7b0f5540-eb44-44a5-b9b1-37d08884a6b0 None None] [instance: bec03c58-ceb4-4e19-8492-779ac61450c0] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid 'bec03c58-ceb4-4e19-8492-779ac61450c0'
2014-03-03T01:38:22.000 2014-03-03 01:25:22.639 ERROR nova.virt.libvirt.driver [req-2ba196b9-c2be-4f27-a5bf-3c19f6bb33a4 AttachInterfacesV3Test-58929740 AttachInterfacesV3Test-599922619] [instance: 1f5a6fc3-a8a7-4183-8c19-0db0cd450d80] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-03-03T01:25:22.000 [req-2ba196b9-c2be-4f27-a5bf-3c19f6bb33a4 AttachInterfacesV3Test-58929740 AttachInterfacesV3Test-599922619] [instance: 1f5a6fc3-a8a7-4183-8c19-0db0cd450d80] Error from libvirt during destroy. Code=38 Error=End of file while reading data: Input/output error
2014-...

Read more...

Changed in nova:
status: Incomplete → New
Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :

I had a brief chat with Daniel Berrangé on #openstack-nova, and like I noted in comment #1, he has this to add:

    [instance: 8e34bcb4-6f64-4a92-b3db-7756f4473c09] Error from libvirt during destroy. Code=42 Error=Domain not found: no domain with matching uuid '8e34bcb4-6f64-4a92-b3db-7756f4473c09'

means:

Nova is trying to tell libvirt to destroy a guest which has already gone. As an error, it is pretty harmless from libvirt's POV, but it might indicate an unexpected scenario from Nova's POV.

Solly Ross (sross-7)
Changed in nova:
importance: Undecided → Low
status: New → Confirmed
Revision history for this message
Joe Gordon (jogo) wrote :
Sean Dague (sdague)
Changed in nova:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.