test_router_interface_ops_bump_router fails with "AssertionError: 5 not greater than 5"

[Ihar Hrachyshka]

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-neutron-python35/neutron/tests/base.py", line 116, in func
    return f(self, *args, **kwargs)
  File "/home/jenkins/workspace/gate-neutron-python35/neutron/tests/unit/services/revisions/test_revision_plugin.py", line 162, in test_router_interface_ops_bump_router
    router['revision_number'])
  File "/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/unittest2/case.py", line 1233, in assertGreater
    self.fail(self._formatMessage(msg, standardMsg))
  File "/home/jenkins/workspace/gate-neutron-python35/.tox/py35/lib/python3.5/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: 5 not greater than 5

http://logs.openstack.org/88/360488/3/gate/gate-neutron-python35/1b33db5/testr_results.html.gz

[Ihar Hrachyshka (ihar-hrachyshka)]

5 hits in 7 days: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%205%20not%20greater%20than%205%5C%22

[Kevin Benton (kevinbenton)]

This may be related to https://review.openstack.org/#/c/452691/

[Kevin Benton (kevinbenton)]

Sorry, I mean that patch may fix the issue

[Reedip (reedip-banerjee-deactivatedaccount)]

Hit me : http://logs.openstack.org/75/442375/17/check/gate-neutron-python35/f005c64/console.html#_2017-04-10_14_59_03_534520

[Ihar Hrachyshka (ihar-hrachyshka)]

Still hits master with the patch included: http://logs.openstack.org/70/454870/1/gate/gate-neutron-python35/6cd99b6/testr_results.html.gz

[Ihar Hrachyshka (ihar-hrachyshka)]

It's interesting that only the last change in the test (removing the subnet interface) seems error prone, it does bump correctly on all other operations like updating name, adding extra routes, adding interface. It also doesn't seem to affect other test cases for other resources like SGs.

[Ihar Hrachyshka (ihar-hrachyshka)]

It's sad that we don't seem to get any logs from neutron code when it happens. I am trying something here: https://review.openstack.org/#/c/456332/ in hope that it will give us neutron logs.

[Ihar Hrachyshka (ihar-hrachyshka)]

<ihrachys> kevinbenton, so the router relationship in RouterPort is the only lazy='dynamic' in the tree. I guess it may be somehow related to https://review.openstack.org/#/c/434454/7/neutron/db/api.py skipping it, but I would need to wrap up my head around the code to understand how.

[Ihar Hrachyshka (ihar-hrachyshka)]

May be relevant: http://docs.sqlalchemy.org/en/latest/orm/collections.html#dynamic-relationship-loaders

"Since the read side of the dynamic relationship always queries the database, changes to the underlying collection will not be visible until the data has been flushed. However, as long as “autoflush” is enabled on the Session in use, this will occur automatically each time the collection is about to emit a query."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/456550

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/456550
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ec03a576143562f2c917276c6aaeebcecdca5788
Submitter: Jenkins
Branch: master

commit ec03a576143562f2c917276c6aaeebcecdca5788
Author: Kevin Benton <email address hidden>
Date: Thu Apr 13 03:15:03 2017 -0700

    Call expire_all in revision tests before re-use

    After the merge of Ia15c63f94d2c67791da3b65546e59f6929c8c685, delete
    port occurs in a separate session. So when an interface is deleted
    from a router, the router object that gets a revision bump is in a
    separate sqlalchemy session.

    This means that if the router object is loaded in a parent session,
    it will not see the updated revision number. This is fine for the normal
    code path since we don't return the revision number of the router after
    the port removal in the API. However, in the unit tests we keep re-using
    the same session for all of the operations so we can get a stale router
    in a get_router call after an interface removal if the router SQLA object
    is still in the identity map of the session.

    The transience of this failure was due to the fact that a stale read
    would only occur if the router SQLA object hadn't been garbage collected.

    This fix alters the unit tests to call expire_all() on the session being
    used whenver it is referenced if it's not in an active transaction.

    Change-Id: I5ba5c5773b85ed9a710e69d7e2f596f760b43fd3
    Closes-Bug: #1679815

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b2 development milestone.

[Ihar Hrachyshka (ihar-hrachyshka)]

The regression was introduced by the new engine facade, which hasn't landed before Pike. Not a backport material.