Cannot commit transaction AddPortCommand exceptions cause tempest network_basic_ops to incur ssh timeouts

Bug #1736284 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Invalid
Critical
Unassigned

Bug Description

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-12-01_14_23_24_962

2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info raise exceptions.NotConnectedError(txn=txn)
2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info NotConnectedError: Cannot commit transaction AddPortCommand(bridge=br-int, may_exist=False, port=qg-d1477f87-c1), DbSetCommand(table=Interface, col_values=(('type', 'internal'), ('external_ids', {'iface-id': u'd1477f87-c182-4dab-ab07-7d170bfe1d04', 'iface-status': 'active', 'attached-mac': u'fa:16:3e:ab:41:bc'})), record=qg-d1477f87-c1). Not connected
2017-12-01 14:23:24.962 68792 ERROR neutron.agent.l3.router_info

Tempest begins to run network_basic_ops around the time of the above failure:

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-12-01_14_23_56

This is not a consistent error atm according to openstack health:

http://status.openstack.org/openstack-health/#/g/build_name/tripleo-ci-centos-7-scenario003-multinode-oooq-container-mistral

Revision history for this message
wes hayutin (weshayutin) wrote :

http://logs.openstack.org/01/511701/7/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/86a2e14/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-12-01_14_23_24_962

2017-12-01 14:23:23.933 68792 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-int'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command [-] Error executing command: NotConnectedError: Cannot commit transaction DelPortCommand(if_exists=True, bridge=None, port=qg-d1477f87-c1). Not connected
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command txn.add(self)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.gen.next()
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self._nested_txn = None
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 52, in commit
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command self.ovsdb_connection.queue_txn(self)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 120, in queue_txn
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command raise exceptions.NotConnectedError(txn=txn)
2017-12-01 14:23:24.254 68792 ERROR ovsdbapp.backend.ovs_idl.command NotConnectedError: Cannot commit transaction DelPortCommand(if_exists=True, bridge=None, port=qg-d1477f87-c1). Not connected

Changed in tripleo:
status: New → Triaged
wes hayutin (weshayutin)
summary: - TestNetworkBasicOps:test_network_basic_ops in tripleo errors with ssh
- timeout
+ Cannot commit transaction AddPortCommand exceptions cause tempest
+ network_basic_ops to incur ssh timeouts
Revision history for this message
Emilien Macchi (emilienm) wrote :

You had python-ovsdbapp-0.6.0-0.20171024151214.0b86d12.el7.centos.src.rpm deployed - with the latest promotion we have python-ovsdbapp-0.9.0-0.20171130203756.7d4b4b6.el7.centos.src.rpm (promotion from today).

I'm closing it since you shouldn't have this problem anymore, but please re-open if you still see it from now.

Changed in tripleo:
status: Triaged → Invalid
Revision history for this message
wes hayutin (weshayutin) wrote :

This job has not failed since Dec 1, removing alert and monitoring the job.. No work is required atm.

tags: removed: alert ci
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers