zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent

[Ryan Beisner]

Gate test failing with "zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent." The charm state is good, quagga has an AS listed.

Observed on xenial-pike, but may not be limited to that.

neutron-dynamic-routing/0* active idle 3 172.17.115.13 Unit is ready

quagga/0* active idle 4 172.17.115.6 Ready (AS Number 65285)

Artifact index:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/index.html

Juju status:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/test_charm_func_full_6934/juju-status-zaza-3680dc9b891b.txt

Juju crashdump:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/test_charm_func_full_6934/juju-crashdump-137bc80d-0c7e-42ee-830c-43e3a8c79313.tar.xz

Test console output:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-dynamic-routing/677906/1/3660/consoleText.test_charm_func_full_6934.txt

2019-08-22 13:05:15 [INFO] Deploying bundle './tests/bundles/xenial-pike-functional.yaml' on to 'zaza-3680dc9b891b' model
2019-08-22 13:05:15 [INFO] Rendered template '<Template memory:7fb690c2eda0>' to file '/tmp/tmp_x05hxui/local-charm-overlay.yaml'
2019-08-22 13:05:15 [INFO] Deploying overlay '/tmp/tmp_x05hxui/local-charm-overlay.yaml' on to 'zaza-3680dc9b891b' model
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/keystone
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/percona-cluster
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/neutron-api
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/quagga
2019-08-22 13:05:16 [INFO] Resolving charm: cs:~openstack-charmers-next/xenial/rabbitmq-server
2019-08-22 13:05:16 [INFO] Executing changes:
2019-08-22 13:05:16 [INFO] - upload charm cs:~openstack-charmers-next/keystone-453 for series xenial
2019-08-22 13:05:17 [INFO] - deploy application keystone on xenial using cs:~openstack-charmers-next/keystone-453
2019-08-22 13:05:18 [INFO] - upload charm cs:~openstack-charmers-next/percona-cluster-353 for series xenial
2019-08-22 13:05:19 [INFO] - deploy application mysql on xenial using cs:~openstack-charmers-next/percona-cluster-353
2019-08-22 13:05:20 [INFO] - upload charm cs:~openstack-charmers-next/neutron-api-413 for series xenial
2019-08-22 13:05:20 [INFO] - deploy application neutron-api on xenial using cs:~openstack-charmers-next/neutron-api-413
2019-08-22 13:05:22 [INFO] - upload charm /var/lib/jenkins/workspace/test_charm_func_full/asset/neutron-dynamic-routing/tests/bundles/../../../neutron-dynamic-routing for series xenial
2019-08-22 13:05:23 [INFO] - deploy application neutron-dynamic-routing on xenial using /var/lib/jenkins/workspace/test_charm_func_full/asset/neutron-dynamic-routing/tests/bundles/../../../neutron-dynamic-routing
2019-08-22 13:05:24 [INFO] - upload charm cs:~openstack-charmers-next/quagga-6 for series xenial
2019-08-22 13:05:26 [INFO] - deploy application quagga on xenial using cs:~openstack-charmers-next/quagga-6
2019-08-22 13:05:27 [INFO] - upload charm cs:~openstack-charmers-next/rabbitmq-server-348 for series xenial
2019-08-22 13:05:27 [INFO] - deploy application rabbitmq-server on xenial using cs:~openstack-charmers-next/rabbitmq-server-348
2019-08-22 13:05:28 [INFO] - add relation neutron-dynamic-routing - rabbitmq-server
2019-08-22 13:05:29 [INFO] - add relation neutron-dynamic-routing - quagga
2019-08-22 13:05:29 [INFO] - add relation keystone - mysql
2019-08-22 13:05:29 [INFO] - add relation neutron-api - mysql
2019-08-22 13:05:29 [INFO] - add relation neutron-api - rabbitmq-server
2019-08-22 13:05:29 [INFO] - add relation neutron-api - keystone
2019-08-22 13:05:29 [INFO] - add unit keystone/0 to new machine 0
2019-08-22 13:05:30 [INFO] - add unit mysql/0 to new machine 1
2019-08-22 13:05:31 [INFO] - add unit neutron-api/0 to new machine 2
2019-08-22 13:05:32 [INFO] - add unit neutron-dynamic-routing/0 to new machine 3
2019-08-22 13:05:34 [INFO] - add unit quagga/0 to new machine 4
2019-08-22 13:05:35 [INFO] - add unit rabbitmq-server/0 to new machine 5
2019-08-22 13:05:36 [INFO] Deploy of bundle completed.
2019-08-22 13:05:36 [INFO] Waiting for environment to settle
2019-08-22 13:05:36 [INFO] Waiting for a unit to appear
2019-08-22 13:05:36 [INFO] Waiting for all units to be idle
2019-08-22 13:17:13 [INFO] Checking workload status of keystone/0
2019-08-22 13:17:13 [INFO] Checking workload status message of keystone/0
2019-08-22 13:17:13 [INFO] Checking workload status of neutron-api/0
2019-08-22 13:17:13 [INFO] Checking workload status message of neutron-api/0
2019-08-22 13:17:13 [INFO] Checking workload status of quagga/0
2019-08-22 13:17:13 [INFO] Checking workload status message of quagga/0
2019-08-22 13:17:13 [INFO] Checking workload status of mysql/0
2019-08-22 13:17:13 [INFO] Checking workload status message of mysql/0
2019-08-22 13:17:13 [INFO] Checking workload status of rabbitmq-server/0
2019-08-22 13:17:13 [INFO] Checking workload status message of rabbitmq-server/0
2019-08-22 13:17:13 [INFO] Checking workload status of neutron-dynamic-routing/0
2019-08-22 13:17:13 [INFO] Checking workload status message of neutron-dynamic-routing/0
2019-08-22 13:17:16 [INFO] Using keystone API V2 for overcloud auth
2019-08-22 13:17:20 [INFO] Configuring overcloud network
2019-08-22 13:17:21 [INFO] Configuring external network
2019-08-22 13:17:21 [INFO] Creating new external network definition: ext_net
2019-08-22 13:17:22 [INFO] New external network created: 3b063a5d-48d5-4445-ad17-7d1d9b222f2c
2019-08-22 13:17:22 [INFO] Creating new subnet
2019-08-22 13:17:22 [INFO] New subnet created: 0637caff-d5ec-4d14-9748-ea163b24f22a
2019-08-22 13:17:22 [INFO] Creating provider router for external network access
2019-08-22 13:17:22 [INFO] New router created: cd13cb6e-f592-4187-8da4-0105d88e148f
2019-08-22 13:17:22 [INFO] Plugging router into ext_net
2019-08-22 13:17:25 [INFO] Router connected
2019-08-22 13:17:25 [INFO] Creating public address scope
2019-08-22 13:17:25 [INFO] New address scope created: cfb0f793-5ddd-49c5-b73b-213c18136dd0
2019-08-22 13:17:25 [INFO] Creating subnetpool: pooled_subnets
2019-08-22 13:17:25 [INFO] Creating network: private
2019-08-22 13:17:25 [INFO] Creating subnet
2019-08-22 13:17:26 [INFO] Updating dns_nameservers (10.5.0.2) for subnet
2019-08-22 13:17:27 [INFO] Adding interface from subnet to provider-router
2019-08-22 13:17:29 [INFO] Adding ssh security group rule
2019-08-22 13:17:29 [INFO] Adding ping security group rule
2019-08-22 13:17:37 [INFO] Setting up BGP speaker
2019-08-22 13:17:37 [INFO] Creating BGP Speaker
2019-08-22 13:17:37 [INFO] Advertising BGP routes
2019-08-22 13:17:37 [INFO] Advertising ext_net network on BGP Speaker bgp-speaker
2019-08-22 13:17:37 [INFO] Advertising private network on BGP Speaker bgp-speaker
2019-08-22 13:17:38 [INFO] Setting up BGP peer
2019-08-22 13:17:38 [INFO] Creating BGP Peer
2019-08-22 13:17:38 [INFO] Adding BGP peer to BGP speaker
2019-08-22 13:17:38 [INFO] Adding peer quagga on BGP Speaker bgp-speaker
2019-08-22 13:17:38 [INFO] Creating floating IP to advertise
2019-08-22 13:17:38 [INFO] Creating port: FIP TEST
2019-08-22 13:17:39 [INFO] Creating floatingip
2019-08-22 13:17:42 [INFO] Advertised floating IP: 172.17.115.200
2019-08-22 13:17:42 [INFO] Waiting for Neutron agent 'neutron-bgp-dragent' to appear...
2019-08-22 13:17:47 [INFO] Using keystone API V2 for overcloud auth
2019-08-22 13:17:50 [INFO] Waiting for BGP speaker to appear on agent "02830902-75ac-4eb3-840d-f10b337294d8"...
Traceback (most recent call last):
  File "/tmp/tmp.zngYgJyU0i/func/bin/functest-run-suite", line 10, in <module>
    sys.exit(main())
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 140, in main
    bundle=args.bundle)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 71, in func_test_runner
    config_steps.get(model_alias, []))
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 48, in configure
    run_configure_list(functions)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/dragent/configure.py", line 92, in setup
    keystone_session=keystone_session)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/configure/bgp_speaker.py", line 119, in setup_bgp_speaker
    neutron_client, agent_id)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 331, in iter
    raise retry_exc.reraise()
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 167, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/tmp/tmp.zngYgJyU0i/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2147, in neutron_bgp_speaker_appears_on_agent
    ''.format(agent_id))
zaza.openstack.utilities.exceptions.NeutronBGPSpeakerMissing: No BGP Speaker appeared on agent "02830902-75ac-4eb3-840d-f10b337294d8"

[Alex Kavanagh (ajkavanagh)]

Setting to incomplete in case this is essentially a one off; please re-open if it occurs again

[Launchpad Janitor (janitor)]

[Expired for charm-neutron-dynamic-routing because there has been no activity for 60 days.]

[David Ames (thedac)]

This bug is in many ways a duplicate of https://bugs.launchpad.net/charm-neutron-dynamic-routing/+bug/1784083. The core root problem is the same.

The root cause seems to be upstream in the dragent. It may never have been envisioned to run the agent by itself the way the charm does.

So that even if neutron-api completes its amqp relation first, eutron-dynamic-routing can still see
    oslo_messaging.exceptions.MessagingTimeout errors. Some operation must occur against neutron before dragent is truly ready. i.e. some post deploy openstack command. So it is outside the purview of the charm.

The following changes add the amqp relation to neutron-dynamic-routing late. Post-deploy after networks have been configured. They are here for a proof of concept and request for comments:

https://github.com/openstack-charmers/zaza-openstack-tests/pull/155
https://review.opendev.org/#/c/703207/

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Ames (<email address hidden>) on branch: master
Review: https://review.opendev.org/703207
Reason: Fixed in the testing suite:
https://github.com/openstack-charmers/zaza-openstack-tests/pull/155

[David Ames (thedac)]

Issue was resolved in the testing suite: https://github.com/openstack-charmers/zaza-openstack-tests/pull/155

The charm may theoretically still have the bug so I will leave this open but remove myself as assignee.