neutron

test_subnet_tags gate failure

Bug #1673243 reported by Kevin Benton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Kevin Benton

Bug Description

http://logs.openstack.org/50/439750/2/gate/gate-neutron-dsvm-api-ubuntu-xenial/d434861/logs/testr_results.html.gz

2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource [req-c03cb69d-aa18-409d-a002-0927d616362b tempest-TagSubnetTestJSON-1826626474 tempest-TagSubnetTestJSON-1826626474] update_all failed: No details.
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/extensions/tag.py", line 130, in update_all
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource parent_id, body)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wrapper
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self.force_reraise()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/services/tag/tag_plugin.py", line 96, in update_tags
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource if tag_db.tag in tags_removed
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/objects/base.py", line 468, in delete_objects
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource context, cls.db_model, **cls.modify_fields_to_db(kwargs))
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/objects/db/api.py", line 102, in delete_objects
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource return len(db_objs)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 553, in __exit__
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self.rollback()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 550, in __exit__
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self.commit()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 455, in commit
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self._prepare_impl()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 435, in _prepare_impl
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self.session.flush()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2080, in flush
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource self._flush(objects)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2198, in _flush
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource transaction.rollback(_capture_exception=True)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2162, in _flush
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource flush_context.execute()
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource rec.execute(self)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource uow
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource mapper, table, update)
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 732, in _emit_update_statements
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource (table.description, len(records), rows))
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource StaleDataError: UPDATE statement on table 'standardattributes' expected to update 4 row(s); 3 were matched.
2017-03-15 16:38:27.076 30202 ERROR neutron.api.v2.resource

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/446145

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Kevin Benton (kevinbenton) wrote :

https://review.openstack.org/#/c/446145/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/446152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/446145
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=44b6fcb4d26eaa5e28f147d903482f8fdd37d911
Submitter: Jenkins
Branch: master

commit 44b6fcb4d26eaa5e28f147d903482f8fdd37d911
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:42:33 2017 -0700

    Use correct retry mechanism in tags

    This replaces wrap_db_retry with
    neutron.db.api.retry_if_session_inactive() in the tags
    plugin.

    wrap_db_retry is missing a bunch of exception types we
    can encounter and are expected to retry in normal database
    operations in Neutron (e.g. staledataerror in the bug report).

    This patch doesn't include a unit test because there is already
    quite a bit of coverage ensuring the decorator handles these types
    of exceptions.

    Closes-Bug: #1673243
    Change-Id: I65b10ee54301401cd4525152a82a604ac8744582

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/446152
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=99e5f4fb4680082407f6c38bad6b7345696e5e9f
Submitter: Jenkins
Branch: master

commit 99e5f4fb4680082407f6c38bad6b7345696e5e9f
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:56:21 2017 -0700

    Get rid of custom wrap_db_retry call in sync_allocations

    Just for consistency, get rid of the last direct call to
    wrap_db_retry and instead use neutron.db.api.retry_db_errors
    in the sync_allocations code so we don't potentially get
    bitten by slight inconsistencies between the two.

    TrivialFix
    Related-Bug: #1673243

    Change-Id: I975c7d0ec84698d277acbbec6d9cfec5c03439aa

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.0.0b1

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/460908

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/460909

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/460911

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/460912

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/460908
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=185d3728352a684c14d97c5100f664ed313f75be
Submitter: Jenkins
Branch: stable/ocata

commit 185d3728352a684c14d97c5100f664ed313f75be
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:42:33 2017 -0700

    Use correct retry mechanism in tags

    This replaces wrap_db_retry with
    neutron.db.api.retry_if_session_inactive() in the tags
    plugin.

    wrap_db_retry is missing a bunch of exception types we
    can encounter and are expected to retry in normal database
    operations in Neutron (e.g. staledataerror in the bug report).

    This patch doesn't include a unit test because there is already
    quite a bit of coverage ensuring the decorator handles these types
    of exceptions.

    Closes-Bug: #1673243
    Change-Id: I65b10ee54301401cd4525152a82a604ac8744582
    (cherry picked from commit 44b6fcb4d26eaa5e28f147d903482f8fdd37d911)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/460909
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=550c7e72783b7b4acac7b2eaab0749a5c06d5ef6
Submitter: Jenkins
Branch: stable/ocata

commit 550c7e72783b7b4acac7b2eaab0749a5c06d5ef6
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:56:21 2017 -0700

    Get rid of custom wrap_db_retry call in sync_allocations

    Just for consistency, get rid of the last direct call to
    wrap_db_retry and instead use neutron.db.api.retry_db_errors
    in the sync_allocations code so we don't potentially get
    bitten by slight inconsistencies between the two.

    TrivialFix
    Related-Bug: #1673243

    Change-Id: I975c7d0ec84698d277acbbec6d9cfec5c03439aa
    (cherry picked from commit 99e5f4fb4680082407f6c38bad6b7345696e5e9f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/460911
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=acb1dfb7632717abac41a61bc81100f2d62c0845
Submitter: Jenkins
Branch: stable/newton

commit acb1dfb7632717abac41a61bc81100f2d62c0845
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:42:33 2017 -0700

    Use correct retry mechanism in tags

    This replaces wrap_db_retry with
    neutron.db.api.retry_if_session_inactive() in the tags
    plugin.

    wrap_db_retry is missing a bunch of exception types we
    can encounter and are expected to retry in normal database
    operations in Neutron (e.g. staledataerror in the bug report).

    This patch doesn't include a unit test because there is already
    quite a bit of coverage ensuring the decorator handles these types
    of exceptions.

    Conflicts:
     neutron/services/tag/tag_plugin.py

    Closes-Bug: #1673243
    Change-Id: I65b10ee54301401cd4525152a82a604ac8744582
    (cherry picked from commit 44b6fcb4d26eaa5e28f147d903482f8fdd37d911)
    (cherry picked from commit 185d3728352a684c14d97c5100f664ed313f75be)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/460912
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3567667f512b87eb5390eb420d496c1d864d2006
Submitter: Jenkins
Branch: stable/newton

commit 3567667f512b87eb5390eb420d496c1d864d2006
Author: Kevin Benton <email address hidden>
Date: Wed Mar 15 12:56:21 2017 -0700

    Get rid of custom wrap_db_retry call in sync_allocations

    Just for consistency, get rid of the last direct call to
    wrap_db_retry and instead use neutron.db.api.retry_db_errors
    in the sync_allocations code so we don't potentially get
    bitten by slight inconsistencies between the two.

    TrivialFix
    Related-Bug: #1673243

    Change-Id: I975c7d0ec84698d277acbbec6d9cfec5c03439aa
    (cherry picked from commit 99e5f4fb4680082407f6c38bad6b7345696e5e9f)
    (cherry picked from commit 9096ea2805a8db07223b05d0f8c805bb568cdef9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.4.0

This issue was fixed in the openstack/neutron 9.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.2

This issue was fixed in the openstack/neutron 10.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.