CIDR, 192.168.0.0/24, not found in BGP peer's routing table

[Ryan Beisner]

Periodically, the functional tests in the charm gate for neutron-dynamic-routing fail with the following (race condition or actual failure?):

2019-10-01 07:19:41 [INFO] ## Running Test zaza.openstack.charm_tests.dragent.tests.DRAgentTest ##
2019-10-01 07:19:41 [INFO] test_bgp_routes (zaza.openstack.charm_tests.dragent.tests.DRAgentTest)
2019-10-01 07:19:41 [INFO] Run bgp tests.
2019-10-01 07:19:41 [INFO] ...
2019-10-01 07:19:45 [INFO] Using keystone API V3 (or later) for overcloud auth
2019-10-01 07:22:01 [INFO] FAIL
2019-10-01 07:22:01 [INFO] ======================================================================
2019-10-01 07:22:01 [INFO] FAIL: test_bgp_routes (zaza.openstack.charm_tests.dragent.tests.DRAgentTest)
2019-10-01 07:22:01 [INFO] Run bgp tests.
2019-10-01 07:22:01 [INFO] ----------------------------------------------------------------------
2019-10-01 07:22:01 [INFO] Traceback (most recent call last):
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/zaza/openstack/charm_tests/dragent/tests.py", line 37, in test_bgp_routes
2019-10-01 07:22:01 [INFO] test.test_bgp_routes(peer_application_name=self.BGP_PEER_APPLICATION)
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/zaza/openstack/charm_tests/dragent/test.py", line 75, in test_bgp_routes
2019-10-01 07:22:01 [INFO] _assert_cidr_in_peer_routing_table(peer_unit, private_cidr)
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/tenacity/__init__.py", line 292, in wrapped_f
2019-10-01 07:22:01 [INFO] return self.call(f, *args, **kw)
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/tenacity/__init__.py", line 358, in call
2019-10-01 07:22:01 [INFO] do = self.iter(retry_state=retry_state)
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/tenacity/__init__.py", line 331, in iter
2019-10-01 07:22:01 [INFO] raise retry_exc.reraise()
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/tenacity/__init__.py", line 167, in reraise
2019-10-01 07:22:01 [INFO] raise self.last_attempt.result()
2019-10-01 07:22:01 [INFO] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
2019-10-01 07:22:01 [INFO] return self.__get_result()
2019-10-01 07:22:01 [INFO] File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
2019-10-01 07:22:01 [INFO] raise self._exception
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/tenacity/__init__.py", line 361, in call
2019-10-01 07:22:01 [INFO] result = fn(*args, **kwargs)
2019-10-01 07:22:01 [INFO] File "/tmp/tmp.ndVf0ILvrN/func-smoke/lib/python3.5/site-packages/zaza/openstack/charm_tests/dragent/test.py", line 73, in _assert_cidr_in_peer_routing_table
2019-10-01 07:22:01 [INFO] "CIDR, {}, not found in BGP peer's routing table" .format(cidr))
2019-10-01 07:22:01 [INFO] AssertionError: CIDR, 192.168.0.0/24, not found in BGP peer's routing table
2019-10-01 07:22:01 [INFO] ----------------------------------------------------------------------
2019-10-01 07:22:01 [INFO] Ran 1 test in 140.269s
2019-10-01 07:22:01 [INFO] FAILED
2019-10-01 07:22:01 [INFO] (failures=1)

Juju application status is healthy on all units.

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-neutron-dynamic-routing/685839/3/12569/index.html

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-neutron-dynamic-routing/685839/3/12569/consoleText.test_charm_func_smoke_14748.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-neutron-dynamic-routing/685839/3/12569/test_charm_func_smoke_14748/juju-status-zaza-284e392c3909.txt

Affects: https://review.opendev.org/#/c/685839/

[Ryan Beisner (1chb1n)]

FYI, additional test-time log info will be available on future runs of this test, and hopefully that will give more info to triage.

Log changes merged @ https://github.com/openstack-charmers/zaza-openstack-tests/pull/100/files

Rechecking @ https://review.opendev.org/#/c/685839

[Aurelien Lourot (aurelien-lourot)]

Seen again:
https://review.opendev.org/c/openstack/charm-neutron-dynamic-routing/+/761574
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-neutron-dynamic-routing/761574/2/20878/consoleText.test_charm_func_smoke_20404.txt