ovs_lib: get_port_tag_dict race with port removal

[YAMAMOTO Takashi]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIlVuYWJsZSB0byBleGVjdXRlIFwiIEFORCBtZXNzYWdlOiBcIi0tY29sdW1ucz1uYW1lLHRhZ1wiIEFORCBtZXNzYWdlOiBcIm92cy12c2N0bDogbm8gcm93XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyOTE1OTgwNzI3OX0=

http://logs.openstack.org/45/160245/42/check/check-tempest-dsvm-neutron-dvr/09be830/logs/screen-q-agt.txt.gz
2015-04-15 10:05:53.275 DEBUG neutron.agent.linux.utils [req-3c5b23d8-3c6f-4145-
bdc2-c21c106b4192 None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list
-ports', 'br-int']
Exit code: 0
Stdin:
Stdout: patch-tun\nqr-5a91d525-f6\nqr-a84affc0-ff\nqvo34a97bc9-9a\nsg-1e0fedbc-a
9\nsg-29d238f1-d0\nsg-591a0885-ba\nsg-d27f8240-60\ntap2566d7d2-51\ntap3704a993-8
5\ntapd8d2b093-c5

Stderr: execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:134
2015-04-15 10:05:53.275 DEBUG neutron.agent.linux.utils [req-3c5b23d8-3c6f-4145-
bdc2-c21c106b4192 None None] Running command (rootwrap daemon): ['ovs-vsctl', '-
-timeout=10', '--oneline', '--format=json', '--', '--columns=name,tag', 'list',
'Port', 'patch-tun', 'qr-5a91d525-f6', 'qr-a84affc0-ff', 'qvo34a97bc9-9a', 'sg-1
e0fedbc-a9', 'sg-29d238f1-d0', 'sg-591a0885-ba', 'sg-d27f8240-60', 'tap2566d7d2-
51', 'tap3704a993-85', 'tapd8d2b093-c5'] execute_rootwrap_daemon /opt/stack/new/
neutron/neutron/agent/linux/utils.py:100
2015-04-15 10:05:53.276 3624 DEBUG neutron.agent.linux.ovsdb_monitor [-] Output
received from ovsdb monitor: {"data":[["b3d74d22-a37d-4cdc-a85c-e14437a46cc8","d
elete","sg-591a0885-ba",75]],"headings":["row","action","name","ofport"]}
 _read_stdout /opt/stack/new/neutron/neutron/agent/linux/ovsdb_monitor.py:44
2015-04-15 10:05:53.280 DEBUG neutron.agent.linux.utils [req-3c5b23d8-3c6f-4145-
bdc2-c21c106b4192 None None]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--co
lumns=name,tag', 'list', 'Port', u'patch-tun', u'qr-5a91d525-f6', u'qr-a84affc0-
ff', u'qvo34a97bc9-9a', u'sg-1e0fedbc-a9', u'sg-29d238f1-d0', u'sg-591a0885-ba',
 u'sg-d27f8240-60', u'tap2566d7d2-51', u'tap3704a993-85', u'tapd8d2b093-c5']
Exit code: 1
Stdin:
Stdout:
Stderr: ovs-vsctl: no row "sg-591a0885-ba" in table Port
 execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:134
2015-04-15 10:05:53.281 ERROR neutron.agent.ovsdb.impl_vsctl [req-3c5b23d8-3c6f-
4145-bdc2-c21c106b4192 None None] Unable to execute ['ovs-vsctl', '--timeout=10'
, '--oneline', '--format=json', '--', '--columns=name,tag', 'list', 'Port', u'pa
tch-tun', u'qr-5a91d525-f6', u'qr-a84affc0-ff', u'qvo34a97bc9-9a', u'sg-1e0fedbc
-a9', u'sg-29d238f1-d0', u'sg-591a0885-ba', u'sg-d27f8240-60', u'tap2566d7d2-51'
, u'tap3704a993-85', u'tapd8d2b093-c5'].
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl Traceback (mos
t recent call last):
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl File "/opt/s
tack/new/neutron/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl log_fail_a
s_error=False).rstrip()
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl File "/opt/s
tack/new/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl raise Runt
imeError(m)
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError:
2015-04-15 10:05:53.281 3624 TRACE neutron.agent.ovsdb.impl_vsctl Command: ['ovs
-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=name,tag
', 'list', 'Port', u'patch-tun', u'qr-5a91d525-f6', u'qr-a84affc0-ff', u'qvo34a9
7bc9-9a', u'sg-1e0fedbc-a9', u'sg-29d238f1-d0', u'sg-591a0885-ba', u'sg-d27f8240
-60', u'tap2566d7d2-51', u'tap3704a993-85', u'tapd8d2b093-c5']

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/174203

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174203
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d36940b720616ec6607c62eca50023eb00bdae01
Submitter: Jenkins
Branch: master

commit d36940b720616ec6607c62eca50023eb00bdae01
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Apr 16 13:39:55 2015 +0900

    ovs_lib: Fix a race between get_port_tag_dict and port removal

    get_port_tag_dict() gets a list of ports using get_port_name_list()
    and then queries the db again for ports in the list.
    It fails if some of ports disappeared in between.
    This change fixes it by ignoring "not exist" errors in the later query.

    Closes-Bug: #1444797
    Change-Id: Ic54b644bb1d72a4664b70f124863d17805c26fff

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/291831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/291831
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e4fb1287d767e4048c4f095f8e1e8a70f11828b3
Submitter: Jenkins
Branch: stable/kilo

commit e4fb1287d767e4048c4f095f8e1e8a70f11828b3
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Apr 16 13:39:55 2015 +0900

    ovs_lib: Fix a race between get_port_tag_dict and port removal

    get_port_tag_dict() gets a list of ports using get_port_name_list()
    and then queries the db again for ports in the list.
    It fails if some of ports disappeared in between.
    This change fixes it by ignoring "not exist" errors in the later query.

    Closes-Bug: #1444797
    Change-Id: Ic54b644bb1d72a4664b70f124863d17805c26fff
    (cherry picked from commit d36940b720616ec6607c62eca50023eb00bdae01)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 2015.1.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 2015.1.4 release.