So I debugged from this failure:
http://logs.openstack.org/40/402140/7/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/ad0cf41/logs/grenade.sh.txt.gz#_2016-12-05_09_07_08_778
The server uuid there is 7e8aad9d-50e1-4b6c-8418-7395a035256f.
I traced that through to n-cpu where nova calls os-terminate_connection in cinder:
http://logs.openstack.org/40/402140/7/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/ad0cf41/logs/new/screen-n-cpu.txt.gz#_2016-12-05_09_07_13_263
Tracing that request to c-api gets me to the point that c-api does an RPC CALL to c-vol:
http://logs.openstack.org/40/402140/7/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/ad0cf41/logs/new/screen-c-api.txt.gz#_2016-12-05_09_07_13_391
At that point it hangs, the RPC call never comes back and we timeout and die. I also don't see the request from c-api req-42b76e49-9bec-4ac5-b457-938bc82b3649 show up in c-vol so it looks like it's just dropped.
So I debugged from this failure:
http:// logs.openstack. org/40/ 402140/ 7/check/ gate-grenade- dsvm-neutron- multinode- ubuntu- xenial/ ad0cf41/ logs/grenade. sh.txt. gz#_2016- 12-05_09_ 07_08_778
The server uuid there is 7e8aad9d- 50e1-4b6c- 8418-7395a03525 6f.
I traced that through to n-cpu where nova calls os-terminate_ connection in cinder:
http:// logs.openstack. org/40/ 402140/ 7/check/ gate-grenade- dsvm-neutron- multinode- ubuntu- xenial/ ad0cf41/ logs/new/ screen- n-cpu.txt. gz#_2016- 12-05_09_ 07_13_263
Tracing that request to c-api gets me to the point that c-api does an RPC CALL to c-vol:
http:// logs.openstack. org/40/ 402140/ 7/check/ gate-grenade- dsvm-neutron- multinode- ubuntu- xenial/ ad0cf41/ logs/new/ screen- c-api.txt. gz#_2016- 12-05_09_ 07_13_391
At that point it hangs, the RPC call never comes back and we timeout and die. I also don't see the request from c-api req-42b76e49- 9bec-4ac5- b457-938bc82b36 49 show up in c-vol so it looks like it's just dropped.