StaleDataError: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.

Bug #1465407 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Kevin Benton

Bug Description

Looks like bug 1405379 is back:

http://logs.openstack.org/94/189594/4/gate/gate-tempest-dsvm-neutron-full/657d5f4/logs/screen-q-svc.txt.gz?level=TRACE#_2015-06-15_11_05_37_846

2015-06-15 11:05:37.846 ERROR oslo_messaging.rpc.dispatcher [req-50d50b84-6aa5-4588-a6bc-1ef346a7b8cd None None] Exception during message handling: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 135, in get_devices_details_list
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher for device in kwargs.pop('devices', [])
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 107, in get_device_details
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher port_context.network.current)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1407, in update_port_status
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher port.port_binding.host)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/plugins/ml2/db.py", line 171, in get_binding_levels
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher order_by(models.PortBindingLevel.level).
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2398, in all
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher return list(self)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2514, in __iter__
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher self.session._autoflush()
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1282, in _autoflush
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher self.flush()
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2004, in flush
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher self._flush(objects)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher transaction.rollback(_capture_exception=True)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher flush_context.execute()
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher rec.execute(self)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher uow
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher mapper, table, update)
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 692, in _emit_update_statements
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher (table.description, len(records), rows))
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher StaleDataError: UPDATE statement on table 'ports' expected to update 1 row(s); 0 were matched.
2015-06-15 11:05:37.846 21067 ERROR oslo_messaging.rpc.dispatcher

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RhbGVEYXRhRXJyb3I6IFVQREFURSBzdGF0ZW1lbnQgb24gdGFibGUgJ3BvcnRzJyBleHBlY3RlZCB0byB1cGRhdGUgMSByb3cocyk7IDAgd2VyZSBtYXRjaGVkLlwiIEFORCB0YWdzOlwic2NyZWVuLXEtc3ZjLnR4dFwiIEFORCBtb2R1bGU6XCJvc2xvX21lc3NhZ2luZy5ycGMuZGlzcGF0Y2hlclwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDM0NDAyNjUxOTM1LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

1858 hits (many duplicate traces) in the last 48 hours, check and gate.

Revision history for this message
Matt Riedemann (mriedem) wrote :

There are a few changes that have merged today that are related to updating ports:

https://review.openstack.org/#/c/190162/

https://review.openstack.org/#/c/190195/

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Matt Riedemann (mriedem) wrote :

I guess this isn't resulting in 100% failure in logstash, just a ton of traces.

There is also a DBDeadlock that happens though which is probably causing the failures:

http://logs.openstack.org/94/189594/4/gate/gate-tempest-dsvm-neutron-full/657d5f4/logs/screen-q-svc.txt.gz?level=TRACE#_2015-06-15_11_06_44_430

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Kevin Benton (kevinbenton) wrote :

This is likely a side effect of the increased API workers patch in devstack.

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/192063

Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (171.5 KiB)

Reviewed: https://review.openstack.org/196920
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7f759c077f8f860c13db92d2ea6b353ef6b70900
Submitter: Jenkins
Branch: feature/pecan

commit 8123144fadd7c5d5e6e56a76ea860512619a2cf6
Author: Moshe Levi <email address hidden>
Date: Sun Jun 28 14:37:14 2015 +0300

    Fix Consolidate sriov agent and driver code

    This patch add mising __init to mech_sriov/mech_driver/
    and update the setup.cfg to the new agent entrypoint

    Trivial Fix

    Change-Id: I53a527081feb78472f496675bbb3c5121d38a14a

commit 8942fccf02e6e179d47582fdb2792a1ca972da21
Author: Assaf Muller <email address hidden>
Date: Mon Jun 29 11:38:51 2015 -0400

    Remove failing SafeFixture tests

    The fixtures 1.3 release attempted to fix the fixtures resource
    leak issue, but failed to do so completely. Our own SafeFixture
    is still needed: The 1.3 release broke our SafeFixture tests,
    but not the usage of SafeFixture itself. This patch removes
    those failing tests for now to unbreak the gate. Jakub reported
    a bug on fixtures 1.3:
    https://bugs.launchpad.net/python-fixtures/+bug/1469759

    We will continue to use SafeFixture until that bug is fixed
    in fixtures, at which point we will be able to require
    fixtures > 1.3.

    Change-Id: I59457c3bb198ff86d5ad55a1e623d008f0034b8f
    Closes-Bug: #1469734

commit 71dffb0a2c1720cd8233a329d32958a0160dd6f5
Author: Kevin Benton <email address hidden>
Date: Mon Jun 29 08:27:41 2015 +0000

    Revert "Removed test_lib module"

    This reverts commit 9a6536de6e1a7fe9b2552adc142e254426b82b6f.

    We pulled all of the plugins out of the tree, many of which still inherit
    from neutron test classes. This change then stated that we no longer
    support testing other plugins. I think this is a bit premature and should
    have been discussed under the subject
    "Neutron plugins can't use neutron plugin unit tests" or something
    similar.

    Change-Id: I68318589f010b731574ea3bfa8df98492bab31fc

commit b20fd81dbd497e058384a0af065dd0f1fdc4c728
Author: Jakub Libosvar <email address hidden>
Date: Fri Jun 5 14:32:51 2015 +0000

    Refactor NetcatTester class

    Following capabilities were added:
       - used transport protocol is passed as a constant instead of bool
       - src port for testing was added
       - connection can be established explicitly
       - change constructor parameters of NetcatTester

    As a part of removing bool for protocol definition
    get_free_namespace_port() was also modified to match the behavior.

    Change-Id: Id2ec322e7f731c05a3754a65411c9a5d8b258126

commit 83e37980dcd0b2bad6d64dd2cb23bcd2891cafca
Author: jingliuqing <email address hidden>
Date: Sat Jun 27 13:41:54 2015 +0800

    Use REST rather than ReST

    Change-Id: I06c9deaab58c5ec13bfeec39fb8fd4b1fe21f42d

commit 1b60df85ba3ad442c2e4e7e52538e1b9a1bf9378
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:34:38 2015 -0700

    Add a double-mock guard to the base test case

    Use mock to patch mock with a check to prevent multiple active
    patches to the...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/qos)

Fix proposed to branch: feature/qos
Review: https://review.openstack.org/197751

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/qos)
Download full text (19.6 KiB)

Reviewed: https://review.openstack.org/197751
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7e60d59a34a7415a0b0185dbaa99bdc84723608
Submitter: Jenkins
Branch: feature/qos

commit 3da491cf5fe629559281507f65f12a0e34eaedf7
Author: Assaf Muller <email address hidden>
Date: Tue Jun 30 13:22:17 2015 -0400

    Disable pylint job

    Disabling pylint until it gets unbroken. Pylint 1.4.1 is using
    logilab-common, which had a release on the 30th, breaking pylint.
    Pylint developers are planning a logilab-common release tomorrow
    which should unbreak pylint once again, at which point I'll
    re-enable pylint.

    Change-Id: I5d8aaab8192168946c2a0b74abc1a56848ca51a2
    Related-Bug: #1470186

commit 2bbfe6f8253659ebf6951b6426ffc446baacd420
Author: Russell Bryant <email address hidden>
Date: Tue May 26 17:07:37 2015 -0400

    Move windows requirements to requirements.txt

    Commit 276028cca26af573c14938255e40c58358eabd4a added these
    requirements to setup.py from a custom build hook. These requirements
    can now be expressed in requirements.txt. We need to move them there
    so that the global requirements sync job can continue to keep setup.py
    in sync with the global version.

    Depends-on: I2369971d306c10dc39a1b89698cec95cf7551d07
    Change-Id: I3c07c279d33f6aed46c3a97dd9ba81251e51429a

commit 21ff82d9d33313bb88e5970c7b1829a65f195d33
Author: Rossella Sblendido <email address hidden>
Date: Fri Dec 5 17:34:23 2014 +0100

    Adds base in-tree functional testing of the ovs_neutron_agent

    Base setup and utility methods for functional testing of the
    OVS L2 agent.

    Partially-Implements: blueprint restructure-l2-agent
    Co-Authored-By: Rossella Sblendido <email address hidden>

    Change-Id: I5b3149b2b8502b9b9a36d3e20d909872cc17f8e8

commit 1ac7581c6b7d343d2ee22e6c562871c0465d9735
Author: Livnat Peer <email address hidden>
Date: Tue Jun 30 16:25:57 2015 +0300

    fix spelling mistakes

    Change-Id: If063f111fa42a6644a1dadc7f0c0b9bbfb359294

commit 9b23617111706ef6a89e8ba45457238acaea26e2
Author: Kevin Benton <email address hidden>
Date: Mon Jun 29 22:24:22 2015 -0700

    Increase ping count on ARP spoof test

    The other IPv4 tests all have a count of 2 to tolerate
    ping failures due to slow ARP response/interface setup/etc.
    This patch increases test_arp_spoof_allowed_address_pairs_0cidr
    to 2 to match.

    Closes-Bug: #1470234
    Change-Id: I82bd8397672194f6162eef5392d4f19d57450552

commit 8123144fadd7c5d5e6e56a76ea860512619a2cf6
Author: Moshe Levi <email address hidden>
Date: Sun Jun 28 14:37:14 2015 +0300

    Fix Consolidate sriov agent and driver code

    This patch add mising __init to mech_sriov/mech_driver/
    and update the setup.cfg to the new agent entrypoint

    Trivial Fix

    Change-Id: I53a527081feb78472f496675bbb3c5121d38a14a

commit 8942fccf02e6e179d47582fdb2792a1ca972da21
Author: Assaf Muller <email address hidden>
Date: Mon Jun 29 11:38:51 2015 -0400

    Remove failing SafeFixture tests

    The fixtures 1.3 release attempted to fix the fixtures resource
    leak issue, but failed to do so ...

tags: added: in-feature-qos
Changed in neutron:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-2 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.