multi-node test causes nova-compute to lockup

[John Garbutt]

Its not very clear whats going on here, but here is the symptom.

One of the nova-compute nodes appears to lock up:
http://logs.openstack.org/67/175067/2/check/check-tempest-dsvm-multinode-full/7a95fb0/logs/screen-n-cpu.txt.gz#_2015-05-29_23_27_48_296
It was just completing the termination of an instance:
http://logs.openstack.org/67/175067/2/check/check-tempest-dsvm-multinode-full/7a95fb0/logs/screen-n-cpu.txt.gz#_2015-05-29_23_27_48_153

This is also seen in the scheduler reporting the node as down:
http://logs.openstack.org/67/175067/2/check/check-tempest-dsvm-multinode-full/7a95fb0/logs/screen-n-sch.txt.gz#_2015-05-29_23_31_02_711

On further inspection it seems like the other nova compute node had just started a migration:
http://logs.openstack.org/67/175067/2/check/check-tempest-dsvm-multinode-full/7a95fb0/logs/subnode-2/screen-n-cpu.txt.gz#_2015-05-29_23_27_48_079

We have had issues in the past where olso.locks can lead to deadlocks, its not totally clear if thats happening here. all the periodic tasks happen in the same greenlet, so you can stop them happening if you hold a lock in an RPC call thats being processed, etc. No idea if thats happening here though.

[Joe Gordon (jogo)]

It looks like the delete operation is coming from tempest. But the command never finishes since the lock do_terminate_instance uses is never released

' Lock "e701630a-e0f0-4228-ac9b-475604ac3479" acquired by "do_terminate_instance"'

http://logs.openstack.org/67/175067/2/check/check-tempest-dsvm-multinode-full/7a95fb0/logs/screen-n-cpu.txt.gz#_2015-05-29_23_27_47_445

[Joe Gordon (jogo)]

Fingerprint: message:"has not been heard from in a while" AND tags:"screen-n-sch.txt" AND build_name:"check-tempest-dsvm-multinode-full"

[Joe Gordon (jogo)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiaGFzIG5vdCBiZWVuIGhlYXJkIGZyb20gaW4gYSB3aGlsZVwiIEFORCB0YWdzOlwic2NyZWVuLW4tc2NoLnR4dFwiIEFORCBidWlsZF9uYW1lOlwiY2hlY2stdGVtcGVzdC1kc3ZtLW11bHRpbm9kZS1mdWxsXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzM1ODY2NzI3NjEsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

[Joe Gordon (jogo)]

After looking into this further, looks this happens on either node in the multinode job, always ending in the same place (an error in delete causing nova-compute to hang).

[John Garbutt (johngarbutt)]

Making this high, because ti blocking making multi-node voting

[Markus Zoeller (markus_z) (mzoeller)]

I'm setting it to "in progress" because "jogo" is set as assignee. Or does the combination "incomplete" + "assignee" have a special meaning?

[Joe Gordon (jogo)]

Attempted to run guru meditation report (by sending a SIGUSR1) to the hung nova-compute but it doesn't respond

[Joe Gordon (jogo)]

next step is to attach gdb and get a stacktrace