Possible race in _wait_for_metadata_provisioned_if_needed()
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
networking-ovn |
New
|
Undecided
|
Lucas Alvares Gomes |
Bug Description
Apparently there's a race in the _wait_for_
Sometimes tempest runs fails with:
Traceback (most recent call last):
File "/usr/local/
match.
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
self.
'NoneType' object is not iterable
It seems that by the time the _wait_for_
Full error logs: http://
Here's one run example: http://
description: | updated |
Changed in networking-ovn: | |
assignee: | nobody → Lucas Alvares Gomes (lucasagomes) |
tags: | added: networking-ovn-proactive-backport-potential |
The issue is not that it doesn't exist yet but that it was deleted a few seconds before that error. Actually, in those logs
For some reason, after the port was deleted in both Neutron and OVN, the mech driver receives various events from ovsdbapp that the port has transitioned to down, then to up and then back to down:
-> to Down:
Oct 27 14:11:45.194651 ubuntu- xenial- rax-ord- 0000524329 neutron- server[ 5590]: INFO networking_ ovn.ml2. mech_driver [None req-9675e1f5- e4ee-4929- 8a9d-7b06b20bcc df None None] OVN reports status down for port: cc17cd96- a9b5-4ccf- af2f-aec9601952 b4 xenial- rax-ord- 0000524329 neutron- server[ 5590]: DEBUG networking_ ovn.ml2. mech_driver [None req-9675e1f5- e4ee-4929- 8a9d-7b06b20bcc df None None] Port not found during OVN status down report: cc17cd96- a9b5-4ccf- af2f-aec9601952 b4 {{(pid=5634) set_port_ status_ down /opt/stack/ new/networking- ovn/networking_ ovn/ml2/ mech_driver. py:631} }
Oct 27 14:11:45.236309 ubuntu-
-> to Up:
Oct 27 14:11:45.236699 ubuntu- xenial- rax-ord- 0000524329 neutron- server[ 5590]: INFO networking_ ovn.ml2. mech_driver [None req-9675e1f5- e4ee-4929- 8a9d-7b06b20bcc df None None] OVN reports status up for port: cc17cd96- a9b5-4ccf- af2f-aec9601952 b4 xenial- rax-ord- 0000524329 neutron- server[ 5590]: DEBUG ovsdbapp. backend. ovs_idl. transaction [-] Running txn command(idx=0): DbListCommand( if_exists= False, records=None, table=Port_Binding, columns=None, row=True) {{(pid=5634) do_commit /usr/local/ lib/python2. 7/dist- packages/ ovsdbapp/ backend/ ovs_idl/ transaction. py:84}}
Oct 27 14:11:45.238874 ubuntu-
Oct 27 14:11:45.239120 ubuntu- xenial- rax-ord- 0000524329 neutron- server[ 5590]: DEBUG ovsdbapp. backend. ovs_idl. transaction [-] Transaction caused no change {{(pid=5634) do_commit /usr/local/ lib/python2. 7/dist- packages/ ovsdbapp/ backend/ ovs_idl/ transaction. py:110} }
Oct 27 14:11:45.244375 ubuntu- xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event [None req-9675e1f5- e4ee-4929- 8a9d-7b06b20bcc df None None] Unexpected exception in notify_loop: TypeError: 'NoneType' object is not iterable xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event Traceback (most recent call last): xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event File "/usr/local/ lib/python2. 7/dist- packages/ ovsdbapp/ event.py" , line 117, in notify_loop xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event match.run(event, row, updates) xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event File "/opt/stack/ new/networking- ovn/networking_ ovn/ovsdb/ ovsdb_monitor. py", line 113, in run xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event self.driver. set_port_ status_ up(row. name) xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event File "/opt/stack/ new/networking- ovn/networking_ ovn/ml2/ mech_driver. py", line 602, in set_port_status_up xenial- rax-ord- 0000524329 neutron- server[ 5590]: ERROR ovsdbapp.event self._wait_ for_metadata_ provisioned_ if_needed( port_i. ..
Oct 27 14:11:45.244568 ubuntu-
Oct 27 14:11:45.244742 ubuntu-
Oct 27 14:11:45.244918 ubuntu-
Oct 27 14:11:45.245081 ubuntu-
Oct 27 14:11:45.245240 ubuntu-
Oct 27 14:11:45.245406 ubuntu-
Oct 27 14:11:45.245568 ubuntu-