Test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details is failing from time to time

[Slawek Kaplonski]

Test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details is failing with error like below:

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/opt/stack/tempest/tempest/scenario/test_network_basic_ops.py", line 636, in test_subnet_details
    self.assertEqual(set(dns_servers), set(servers),
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 415, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 502, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: {'1.2.3.4'} != set(): Looking for servers: ['1.2.3.4']. Retrieved DNS nameservers: [] From host: 172.24.5.27.

Example of the failure: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_567/785895/1/gate/neutron-tempest-slow-py3/567fc7f/testr_results.html

[Lajos Katona (lajos-katona)]

A fix for similar issue was merged to tempest:
https://review.opendev.org/c/openstack/tempest/+/779756

That patch fixed the case when default routes already existed in resolv.conf, but this bug for empty resolv.conf

[Lajos Katona (lajos-katona)]

I tried to check this from cirros perspective, but I can't find bugreport or note about such (or any) issue.

However an interesting sentence:
https://opendev.org/openstack/upstream-institute-virtual-environment/src/branch/master/README.md#getting-around-in-the-vm

Bullet10: "DNS might not work in the cirros vm, in this case: echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf`"

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

In this test we are using Ubuntu image, if I'm not wrong:
[ 0.000000] Linux version 5.3.0-26-generic (buildd@lgw01-amd64-039) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 (Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13)

IMO, we should implement a more robust method to check the system DNS. In Ubuntu (and other advance images), we can use "resolvectl". For example, when I add two DNS to the subnet:

* Before renewing the IP
root@ubuntu:~# resolvectl dns
Global:
Link 2 (ens3): 10.20.0.2 10.20.0.3

* Update the subnet DNS nameservers
(overcloud) [stack@undercloud-0 ~]$ openstack subnet set --dns-nameserver 3.3.3.3 subnet1
(overcloud) [stack@undercloud-0 ~]$ openstack subnet set --dns-nameserver 3.3.4.4 subnet1

* Refresh the IP in the VM
root@ubuntu:~# dhclient -r && dhclient
root@ubuntu:~# resolvectl dns
Global: 3.3.4.4 3.3.3.3
Link 2 (ens3): 10.20.0.2 10.20.0.3

We can parse the command exit and use those values.

Regards.

[Lajos Katona (lajos-katona)]

Proposed patch: https://review.opendev.org/c/openstack/tempest/+/794776 don't know why launchpad/gerrit haven't added it here