occational FT failure with "Row removed from DB"

Bug #1475253 reported by YAMAMOTO Takashi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Committed
Undecided
YAMAMOTO Takashi

Bug Description

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIlJ1bnRpbWVFcnJvcjogUm93IHJlbW92ZWQgZnJvbSBEQiBkdXJpbmcgbGlzdGluZy4gUmVxdWVzdCBpbmZvOiBUYWJsZT1Qb3J0LiBDb2x1bW5zPVsnbmFtZScsICdvdGhlcl9jb25maWcnLCAndGFnJ10uIFJlY29yZHM9Tm9uZS5cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzNzA0Mzk3MDE3NH0=

ft1.48: neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_assert_bridges_ports_vxlan(native)_StringException: Empty attachments:
  pythonlogging:'neutron.api.extensions'
  stderr
  stdout

pythonlogging:'': {{{
2015-07-16 05:45:28,544 INFO [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Mapping physical network physnet to bridge br-int746017284
2015-07-16 05:45:28,582 WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int746017284 exceeds the 15 character limitation. It was shortened to int-br-inc6274e to fit.
2015-07-16 05:45:28,582 WARNING [neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent] Creating an interface named br-int746017284 exceeds the 15 character limitation. It was shortened to phy-br-inc6274e to fit.
2015-07-16 05:45:29,206 ERROR [neutron.agent.ovsdb.impl_idl] Traceback (most recent call last):
  File "neutron/agent/ovsdb/native/connection.py", line 84, in run
    txn.results.put(txn.do_commit())
  File "neutron/agent/ovsdb/impl_idl.py", line 92, in do_commit
    ctx.reraise = False
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "neutron/agent/ovsdb/impl_idl.py", line 87, in do_commit
    command.run_idl(txn)
  File "neutron/agent/ovsdb/native/commands.py", line 389, in run_idl
    "Records=%(records)s.") % self.requested_info)
RuntimeError: Row removed from DB during listing. Request info: Table=Port. Columns=['name', 'other_config', 'tag']. Records=None.

2015-07-16 05:45:29,207 ERROR [neutron.agent.ovsdb.native.commands] Error executing command
Traceback (most recent call last):
  File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
    txn.add(self)
  File "neutron/agent/ovsdb/api.py", line 70, in __exit__
    self.result = self.commit()
  File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
    raise result.ex
RuntimeError: Row removed from DB during listing. Request info: Table=Port. Columns=['name', 'other_config', 'tag']. Records=None.
}}}

Traceback (most recent call last):
  File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 284, in test_assert_bridges_ports_vxlan
    agent = self.create_agent()
  File "neutron/tests/functional/agent/test_l2_ovs_agent.py", line 112, in create_agent
    conf=self.config)
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 281, in __init__
    self._restore_local_vlan_map()
  File "neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 316, in _restore_local_vlan_map
    "Port", columns=["name", "other_config", "tag"])
  File "neutron/agent/common/ovs_lib.py", line 148, in db_list
    execute(check_error=check_error, log_errors=log_errors))
  File "neutron/agent/ovsdb/native/commands.py", line 42, in execute
    ctx.reraise = False
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "neutron/agent/ovsdb/native/commands.py", line 35, in execute
    txn.add(self)
  File "neutron/agent/ovsdb/api.py", line 70, in __exit__
    self.result = self.commit()
  File "neutron/agent/ovsdb/impl_idl.py", line 70, in commit
    raise result.ex
RuntimeError: Row removed from DB during listing. Request info: Table=Port. Columns=['name', 'other_config', 'tag']. Records=None.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/202505

Changed in neutron:
assignee: nobody → YAMAMOTO Takashi (yamamoto)
status: New → In Progress
Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

it seems i filed two copies of the same bug (1475252, 1475253)
sorry!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/202505
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=73207ddd5e8b4be800b2d1603d73cabad30423ec
Submitter: Jenkins
Branch: master

commit 73207ddd5e8b4be800b2d1603d73cabad30423ec
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Jul 16 19:56:21 2015 +0900

    ovs_lib: Fix native implementation of db_list

    Move the code to list records into the transaction.

    Closes-Bug: #1473038
    Closes-Bug: #1473049
    Closes-Bug: #1475253
    Change-Id: I9f367f5ba1ecc50b9bb94c6f52507033d0139611

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/207903

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (70.7 KiB)

Reviewed: https://review.openstack.org/207903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9badcd249dab2d3330f0cd608496b59c9f44499a
Submitter: Jenkins
Branch: feature/pecan

commit 991ea00e6c115343eabecc62e86072175823f81f
Author: Moshe Levi <email address hidden>
Date: Thu Jul 30 12:04:15 2015 +0300

    SR-IOV: Fix SR-IOV agent to run ip link commands as root

    Commit https://review.openstack.org/#/c/155523/
    remove the remaining root_helper args, but didn't
    update the SR-IOV agent to execute them as root.

    This patch updates the agent to execute ip link commands
    as root and pass options argument as a list in the
    self._as_root method.

    Closes-Bug: #1479694
    Change-Id: I53cafd61845a69fae3a759fb7526950d655ffa20

commit 5b3bacedf6c014815bef03c2e821b5eb8ef92dcb
Author: Saksham Varma <email address hidden>
Date: Thu Jul 23 22:46:23 2015 -0700

    Moved out cisco n1kv mech driver and db models

    Moving out Cisco N1Kv database models and the mech driver
    to the openstack/networking-cisco repo as part of the second
    phase vendor-core decomposition

    Partial-Bug: #1479123
    Partial-implements: blueprint core-vendor-decomposition

    Change-Id: I65a704b238d8cbe2951a6912fa4f1e8518c6f412

commit 65ac8cd0a80548e115b8ec1d4cfd47a94422fddf
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Jul 29 20:44:01 2015 +0000

    Updated from global requirements

    Change-Id: I3a884a73d51df0f93b5cc88b3e3250d81bb1f455

commit f0e8356d04d04600c072a1c0f0bdf274ff19ab8c
Author: sridhargaddam <email address hidden>
Date: Mon Jul 27 03:46:48 2015 +0000

    Update dhcp agent cache for network:dhcp ports

    When a network with a dhcp_enabled subnet is scheduled on a dhcp
    agent, dhcp driver creates the network:dhcp port for the subnet.
    However, the port info is not updated in dhcp agents internal cache.
    Subsequently if the user deletes the network:dhcp port, the port is
    properly deleted on the server side (i.e., in the database) and when
    the port_delete_end notification is sent to the dhcp agent, it simply
    ignores it as the port entry would be missing in the cache. This patch
    fixes this issue by updating the dhcp agents cache when dhcp driver
    creates the network:dhcp port for the subnets.

    Closes-Bug: #1478426
    Change-Id: I69f5834dd964a4320c606c4e0aa2cdba70416943

commit cb60d0bb4e0cc0cba68f59fdf5f4e89d6ec52950
Author: changzhi <email address hidden>
Date: Thu Jul 16 10:14:16 2015 +0800

    Keep dns nameserver order consistency

    Currently, there is no dns servers prioritization for subnets
    for Neutron.

    Generally speaking, it is useful to keep the order of dns
    nameservers consistent. Add a new column named 'order' in table
    'dnsnameservers' and add nameserver into DB one by one.

    Closes-Bug: #1218629
    Implements: blueprint keep-dns-nameserver-orderconsistency
    Change-Id: Id937aea411397d39370368a4eb45be26c4eefa9e

commit b39e1469e824bc8bc79e1ecafa98825a94811c0b
Author: Salvatore Orlando <email address hidden>
Date: Tue Jun 23 04:54:2...

tags: added: in-feature-pecan
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.