lock wait timeout in update_port_status

[Salvatore Orlando]

There have been several occurences of this bug in check/gate queues.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiKE9wZXJhdGlvbmFsRXJyb3IpICgxMjA1LCAnTG9jayB3YWl0IHRpbWVvdXQgZXhjZWVkZWQ7IHRyeSByZXN0YXJ0aW5nIHRyYW5zYWN0aW9uJylcIiAgQU5EIG1lc3NhZ2U6XCJVUERBVEUgcG9ydHMgU0VUIHN0YXR1c1wiIEFORCBOT1QgbWVzc2FnZTpcIlRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0XCIgQU5EIHRhZ3M6XCJzY3JlZW4tcS1zdmMudHh0XCIgQU5EIE5PVCBidWlsZF9uYW1lOlwiY2hlY2stdGVtcGVzdC1tYXN0ZXItZHN2bS1uZXV0cm9uLWhhdmFuYVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk4NDM5MjQ0NDY2fQ==

266 hits in 7 days at the time of bug report, excluding havana jobs (which are susceptible to lock wait timeout errors anyway because of fixes not backportable from icehouse)
Build failure rate: 20.6%
Hits in gate queue: 26
Failures in gate queue: 1

Notes:
1) Occurrences and failure rate in gate queue are lower because most of the failure happen with the full job, which is not yet voting.
2) Even if failure rate is generally low, a lock wait timeout should be always considered an error, regardless of the outcome of the build job.
3) A detailed look at the logs reveals a pattern similar to bug 1283522, whose fingerprint is being matched. It seems the semaphore lock is ignored, but lockutils lacks the necessary logging to reveal whether a semaphore has been released or not; more investigations are in progress.

[Eugene Nikanorov (enikanorov)]

Is this still seen in the gate?
Looks like Importance can be lowered.

[Elena Ezhova (eezhova)]

This bug seems to be interconnected with https://bugs.launchpad.net/neutron/+bug/1298355

The failure happens when one test tries to delete VIP and its port while some other process tries to update the status of this port and both of them get semaphore "db-access" lock. [1]

The bug may be fixed by https://review.openstack.org/#/c/100934/8

[1] http://logs.openstack.org/13/92013/3/check/check-tempest-dsvm-neutron-2/dc8b822/logs/screen-q-svc.txt.gz?#_2014-06-30_11_40_48_293

[Eugene Nikanorov (enikanorov)]

Marking as inclomplete as this has not been seen for more than a month

[Mehdi Abaakouk (sileht)]

It steal occurs on icehouse branch: http://logs.openstack.org/73/138973/4/check/gate-tempest-dsvm-neutron-src-oslo.messaging-icehouse/db5316a/logs/screen-q-svc.txt.gz?level=TRACE

[Armando Migliaccio (armando-migliaccio)]

This bug is > 172 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]