Frequent UT failure in the ASR1K test case test_router_update_unset_msn_gw_dt

[Sandhya Dasu]

The following test case fails 2/3rds of the time during Ocala UT test runs.
networking_cisco.tests.unit.cisco.l3.test_asr1k_routertype_driver.Asr1kHARouterTypeDriverTestCase

ft40.39: networking_cisco.tests.unit.cisco.l3.test_asr1k_routertype_driver.Asr1kHARouterTypeDriverTestCase.test_router_update_unset_msn_gw_dt_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
WARNING [networking_cisco.plugins.cisco.l3.drivers.asr1k.asr1k_routertype_driver] Global router:Global-router-0000-000000000001[id:5169ee1f-233e-4f3b-a0db-e38dec953a71] is present for hosting device:00000000-0000-0000-0000-000000000001 but there are no tenant or redundancy routers with gateway set on that hosting device. Proceeding to delete global router.
 WARNING [networking_cisco.plugins.cisco.l3.drivers.asr1k.asr1k_routertype_driver] Router 5169ee1f-233e-4f3b-a0db-e38dec953a71 could not be found
 WARNING [networking_cisco.plugins.cisco.l3.drivers.asr1k.asr1k_routertype_driver] Global router:Global-router-0000-000000000002[id:052a2784-f049-4388-90e1-d453540b8cfb] is present for hosting device:None but there are no tenant or redundancy routers with gateway set on that hosting device. Proceeding to delete global router.
 WARNING [networking_cisco.plugins.cisco.l3.drivers.asr1k.asr1k_routertype_driver] Router 052a2784-f049-4388-90e1-d453540b8cfb could not be found
}}}

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-networking-cisco-tox-ocata-ubuntu-xenial/.tox/ocata/local/lib/python2.7/site-packages/neutron/tests/base.py", line 116, in func
    return f(self, *args, **kwargs)
  File "/home/jenkins/workspace/gate-networking-cisco-tox-ocata-ubuntu-xenial/.tox/ocata/local/lib/python2.7/site-packages/neutron/tests/base.py", line 116, in func
    return f(self, *args, **kwargs)
  File "networking_cisco/tests/unit/cisco/l3/test_asr1k_routertype_driver.py", line 672, in test_router_update_unset_msn_gw_dt
    self._test_router_update_unset_msn_gw(same_tenant=False)
  File "networking_cisco/tests/unit/cisco/l3/test_asr1k_routertype_driver.py", line 664, in _test_router_update_unset_msn_gw
    self._update('routers', r2['id'], r_spec)
  File "/home/jenkins/workspace/gate-networking-cisco-tox-ocata-ubuntu-xenial/.tox/ocata/local/lib/python2.7/site-packages/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 547, in _update
    self.assertEqual(expected_code, res.status_int)
  File "/home/jenkins/workspace/gate-networking-cisco-tox-ocata-ubuntu-xenial/.tox/ocata/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/jenkins/workspace/gate-networking-cisco-tox-ocata-ubuntu-xenial/.tox/ocata/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 200 != 404

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/473790

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/473790
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=0ac6a2ec0abe85c94a08a47b242e1374aad74fa6
Submitter: Jenkins
Branch: master

commit 0ac6a2ec0abe85c94a08a47b242e1374aad74fa6
Author: Bob Melander <email address hidden>
Date: Tue Jun 13 13:27:07 2017 +0200

    Workaround failing UT for asr1k router type driver

    This patch disables the pre_backlog_processing function during
    the asr1k_routertype_driver unit tests as it is causing
    intermittent failures. Once there is a root cause fix for the
    failures this workaround will be removed.

    Partial-Bug: #1676435

    Change-Id: I6ced1d98184089d62f76856e5ca84fbb42164ac9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/475462

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475462
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=46261c24e607e6b1fbb17049f2706706853345ff
Submitter: Jenkins
Branch: master

commit 46261c24e607e6b1fbb17049f2706706853345ff
Author: Bob Melander <email address hidden>
Date: Mon Jun 19 18:52:58 2017 +0200

    Fixes incorrect format strings in log statements

    One of the erroneous format strings in particular has been causing
    intermittent failures for some of the ASR1k router type driver
    unit tests. This patch therefore removes the workaround for those
    UT failures provided in commit 0ac6a2ec0abe85c94a08a47b242e1374aad74fa6.

    AKA: DE3463

    Partial-Bug: #1676435

    Change-Id: Ifa9fa8ae8ca2f88e7339ac6aaf8b6adb56a80268

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/480565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/480565
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=22ca4a276d8644f10d4bc3c4a005d2a6c0fa5bc4
Submitter: Jenkins
Branch: master

commit 22ca4a276d8644f10d4bc3c4a005d2a6c0fa5bc4
Author: Bob Melander <email address hidden>
Date: Wed Jul 5 14:41:22 2017 +0200

    Disables return code check in REST API update calls from some unit tests

    As a temporary workaround for bug #1676435 this patch disables the return
    code check in the REST API update call for some ASR1k router type driver
    unit tests. This will reduce the number of Jenkins failures while the
    root caused of the unit test failures are investigated.

    AKA: DE3463

    Partial-Bug: #1676435

    Change-Id: Ia411e2a9a3fce9fee9ed598357aef377c504592e