NeutronNetworkingTest.test_instances_have_networking fails

[Aurelien Lourot]

Seems to be happening always on groovy-victoria but also seen once on bionic-train:

https://review.opendev.org/c/openstack/charm-neutron-gateway/+/761575
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-gateway/761575/2/7704/index.html

https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/761576
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-openvswitch/761576/2/7703/index.html

ERROR: test_instances_have_networking (zaza.openstack.charm_tests.neutron.tests.NeutronNetworkingTest)
Validate North/South and East/West networking.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 905, in test_instances_have_networking
    self.check_connectivity(instance_1, instance_2)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 818, in check_connectivity
    instance_1, instance_2, verify, mtu_min)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/tenacity/__init__.py", line 333, in wrapped_f
    return self(f, *args, **kw)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/tenacity/__init__.py", line 372, in iter
    raise retry_exc.reraise()
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/tenacity/__init__.py", line 189, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/neutron/tests.py", line 679, in validate_instance_can_reach_other
    password=password, privkey=privkey, verify=verify)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2515, in ssh_command
    ssh.connect(ip, username=username, password=None, pkey=key)
  File "/tmp/tmp.tDGtSGrESG/func/lib/python3.5/site-packages/paramiko/client.py", line 368, in connect
    raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.17.112.223

[Aurelien Lourot (aurelien-lourot)]

Seen on bionic-train:

https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/770410
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-api-plugin-ovn/770410/1/7762/index.html

[Aurelien Lourot (aurelien-lourot)]

2021-01-13 17:11:23.228 18911 ERROR nova.conductor.manager [req-de8383b4-31f1-4c50-b828-0c9a81f68c9f 26eb661ec2b243709e49c6b3015aef62 fda1361cf738449b95a3d23efd411856 - fdc974c37d624e649e5bb859fc9bb5b7 fdc974c37d624e649e5bb859fc9bb5b7] Failed to schedule instances: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.

Indeed at that time one of both nova-compute units is blocked with "Services not running that should be: nova-compute", so there isn't enough space for creating the second instance.

nova-compute.log shows

2021-01-13 17:03:48.041 16082 ERROR oslo_service.service oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 021e9616aaaf482fa40f6e104c925d56

So this particular occurrence seems to be a duplicate of lp:1861094

[0]: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-api-plugin-ovn/770410/1/7762/index.html

[Aurelien Lourot (aurelien-lourot)]

^ this was for the last linked failure [0]. There the test succeeds at launching zaza-neutrontests-ins-1 but then fails at launching zaza-neutrontests-ins-2 with the posted nova.conductor.manager error.

[Aurelien Lourot (aurelien-lourot)]

In this occurence of the issue [1] the test can create an instance but can't SSH to it. It seems like several units have been paused but not resumed, and neutron-ovn-metadata-agent isn't running on the ovn-chassis units. This is a duplicate of https://bugs.launchpad.net/charm-ovn-chassis/+bug/1912471

[1]: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-gateway/761575/2/7704/index.html

[Aurelien Lourot (aurelien-lourot)]

Same for [2], so I'm definitely closing this bug, which is one same symptom of several other bugs: lp:1861094, lp:1912471 and lp:1911902

[2]: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-neutron-openvswitch/761576/2/7703/index.html