Comment 14 for bug 1651678

Revision history for this message
Vasyl Saienko (vsaienko) wrote :

@Alex: each Ironic node appear in nova like hypervisor, nova lock whole compute when update resources for hypervisor. In the log [0] you can see that resources are updated for different ironic nodes (nova hypervisors). Short report from the log [2]. When we send 0 values to placement API, it returns 400 error, and it takes near 6 seconds to complete this operation. In [2] I've copied update resources for single node (acquired lock at: 13:24:44.599 released lock at: 13:24:51.281, held 6.681s). So indeed it looks like lock starvation which is caused by trying to send 0 values to placement API, which anyways not valid value and request is discarded on API anyway (we got 400). The fix proposed [3] avoids sending 0 values, as they will never be accepted by API.

[0] http://logs.openstack.org/39/404239/14/check/gate-tempest-dsvm-ironic-ipa-wholedisk-pxe_snmp-tinyipa-ubuntu-xenial-nv/8f98498/logs/screen-n-cpu.txt.gz#_2016-12-21_13_24_24_327
[1] http://paste.openstack.org/show/593421/
[2] http://paste.openstack.org/show/593422/
[3] https://review.openstack.org/#/c/414214/