StaleDataError while updating ml2_dvr_port_bindings

[Armando Migliaccio]

This stacktrace has been observed for a while:

http://logs.openstack.org/17/120917/5/experimental/check-tempest-dsvm-neutron-dvr/960bdbb/logs/screen-q-svc.txt.gz?level=TRACE#_2014-09-16_19_06_14_290

The logstash query is here:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RhbGVEYXRhRXJyb3IgVVBEQVRFIHN0YXRlbWVudCBvbiB0YWJsZSAnbWwyX2R2cl9wb3J0X2JpbmRpbmdzJ1wiIEFORCBtZXNzYWdlOlwiaW4gdXBkYXRlX2R2cl9wb3J0X2JpbmRpbmdcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1xLXN2Yy50eHRcIiBBTkQgYnVpbGRfYnJhbmNoOlwibWFzdGVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0wOS0xNlQxOTowMzozMSswMDowMCIsInRvIjoiMjAxNC0wOS0xNlQxOTowNzoyMSswMDowMCIsInVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQxMTU4OTY1OTE2NywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

However, we need to establish if fix for bug #1371696 struck two birds with one stone.

[Wei Wang (damon-devops)]

Is this related with bug #1381536?

[Armando Migliaccio (armando-migliaccio)]

@Wei Wang: there are multiple issues that manifest themselves in the same way, so the traceback can be the same, but the underlying problem may be different.

[Armando Migliaccio (armando-migliaccio)]

If my understanding is correct, this issue manifests itself when there is an interleaving router delete while binding a port:

An analysis from the logs [1] revealed these sequence of actions:

It's shown that this issue manifests itself when:

a) _ensure_host_set_on_ports is called on a router (in the example: 9706e742-84e2-4fed-b040-c577dd5622cb)
b) the router is deleted shortly after (in around 2014-12-24_00_40_32_020)
c) ports get disposed
d) the bind operations comes afterwards (in around 2014-12-24_00_40_32_607)

[1] http://logs.openstack.org/76/134676/13/check/check-tempest-dsvm-neutron-dvr/caa7479/logs/screen-q-svc.txt.gz?#_2014-12-24_00_40_31_113

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/143761

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/143761
Reason: It seems like the issue no longer appears, which makes this change pointless.

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RhbGVEYXRhRXJyb3IgVVBEQVRFIHN0YXRlbWVudCBvbiB0YWJsZSAnbWwyX2R2cl9wb3J0X2JpbmRpbmdzJ1wiIEFORCBtZXNzYWdlOlwiaW4gdXBkYXRlX2R2cl9wb3J0X2JpbmRpbmdcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1xLXN2Yy50eHRcIiBBTkQgYnVpbGRfYnJhbmNoOlwibWFzdGVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0wOS0xNlQxOTowMzozMSswMDowMCIsInRvIjoiMjAxNC0wOS0xNlQxOTowNzoyMSswMDowMCIsInVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyMzE0OTM4OTA5MywibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Armando Migliaccio (armando-migliaccio)]

It's likely that https://review.openstack.org/#/c/151913/ killed the issue

[Armando Migliaccio (armando-migliaccio)]

It can no longer be verified.

[Zengfa Gao (zfgao)]

For bug 1398446, we saw StaleDataError with ml2_dvr_port_bindings too. I cannot reproduce it either. When I was able to reproduce, it's several compute nodes, we create/remove instance/floatingIP at the same times.

[Ryan Moats (rmoats)]

A check of the logstash query above shows that there were 79 hits in the last 7 days, so marking this bug as confirmed (in the sense that it's been seen again)

[Ryan Moats (rmoats)]

all of the above hits referred to come from the dvr-multinode-full job, but the interesting thing is that they don't cause the job to fail... from the 79 hits, most had a success result from the job

[Ryan Moats (rmoats)]

new logstash query to use: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%20%5C%22StaleDataError%5C%22%20

[Ryan Moats (rmoats)]

Not seeing any occurance of this from a DVR job in logstash over the last 7 days - I think we can mark this as fix released...

[Armando Migliaccio (armando-migliaccio)]

We never merged anything to fix the issue, the issue did come and go; as of now it's not there...so I'd rather mark it invalid.