Cannot commit transaction AddPortCommand exceptions cause tempest network_basic_ops to incur ssh timeouts

[wes hayutin]

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-12-01_14_23_24_962

2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info raise exceptions.NotConnectedError(txn=txn)
2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info NotConnectedError: Cannot commit transaction AddPortCommand(bridge=br-int, may_exist=False, port=qg-d1477f87-c1), DbSetCommand(table=Interface, col_values=(('type', 'internal'), ('external_ids', {'iface-id': u'd1477f87-c182-4dab-ab07-7d170bfe1d04', 'iface-status': 'active', 'attached-mac': u'fa:16:3e:ab:41:bc'})), record=qg-d1477f87-c1). Not connected
2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info

Tempest begins to run network_basic_ops around the time of the above failure:

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-12-01_14_23_56

This is not a consistent error atm according to openstack health:

http://status.openstack.org/openstack-health/#/g/build_name/tripleo-ci-centos-7-scenario003-multinode-oooq-container-mistral

[wes hayutin (weshayutin)]

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-12-01_14_23_24_962

2017-12-01 14:23:23.933 68792 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-int'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: NotConnectedError: Cannot commit transaction DelPortCommand(if_exists=True, bridge=None, port=qg-d1477f87-c1). Not connected
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command txn.add(self)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.gen.next()
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self._nested_txn = None
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 52, in commit
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.ovsdb_connection.queue_txn(self)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 120, in queue_txn
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command raise exceptions.NotConnectedError(txn=txn)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command NotConnectedError: Cannot commit transaction DelPortCommand(if_exists=True, bridge=None, port=qg-d1477f87-c1). Not connected

[Emilien Macchi (emilienm)]

You had python-ovsdbapp-0.6.0-0.20171024151214.0b86d12.el7.centos.src.rpm deployed - with the latest promotion we have python-ovsdbapp-0.9.0-0.20171130203756.7d4b4b6.el7.centos.src.rpm (promotion from today).

I'm closing it since you shouldn't have this problem anymore, but please re-open if you still see it from now.

[wes hayutin (weshayutin)]

This job has not failed since Dec 1, removing alert and monitoring the job.. No work is required atm.