Incorrect Tempest test 'test_dhcpv6_64_subnets' - this test failed because we have more than 1 DHCP agent

[Timur Nurlygayanov]

This bug was reproduced on MOS 6.1 ISO #361.

Note:
This test passed for MOS 6.1 ISO #317 and #345 and failed for MOS 6.1 ISO #361

Steps To Reproduce:
1. Deploy OpenStack cloud with MOS 6.1 with the following configuration: Ubuntu, HA mode; Neutron with GRE; Ceph volumes, images; Sahara, Murano, Ceilo enabled
2. Run Tempest test 'test_dhcpv6_64_subnets' using mos-tempest-runner (http://www.youtube.com/watch?v=0ffKJPrpnE4) or Rally verify.

Expected Result:
Test will pass

Observed Result:
Test failed with the error:
Traceback (most recent call last):
File "/home/developer/mos-tempest-runner/tempest/tempest/api/network/test_dhcp_ipv6.py", line 263, in test_dhcpv6_64_subnets
real_dhcp_ip}.issubset([eui_ip] + dhcp_ip))
File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/unittest2/case.py", line 678, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true

Example of result: https://mirantis.testrail.com/index.php?/tests/view/386485&group_by=cases:section_id&group_order=asc&group_id=1560

[Timur Nurlygayanov (tnurlygayanov)]

Code of test:
https://github.com/openstack/tempest/blob/2af1f21ea8d6ebb9dac926b8932b092efdc92258/tempest/api/network/test_dhcp_ipv6.py#L225-273

Looks like this test checks DHCP v 6, is it works in MOS 6.1?

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced on environments with Neuton VLANs and GRE as well.

[Timur Nurlygayanov (tnurlygayanov)]

This test passed for MOS 6.1 ISO #317 and failed for MOS 6.1 ISO #345 and #361

[Oleg Bondarev (obondarev)]

The problem here is that a bunch of tests in tempest/api/network/test_dhcp_ipv6.py (three that I noticed, test_dhcpv6_64_subnets is one of such) have wrong assumption on which exact ip a port will receive on the dhcp enabled subnet. This is wrong as in theory Neutron IPAM system may assign _any_ ip from the subnet allocation pool based on some internal logic. So the check should be that port got _any_ ip from the allocation pool.

In this particular tests the assumption is that port will receive first or second (if dhcp port takes the first one) ip address from allocation pool [1]. Since now we schedule network to 2 dhcp agents - 2 dhcp ports are created (while the test expects maximum one dhcp port) and port created by test may get only the third ip from allocation pool .

[1] https://github.com/openstack/tempest/blob/2af1f21ea8d6ebb9dac926b8932b092efdc92258/tempest/api/network/test_dhcp_ipv6.py#L254

[Timur Nurlygayanov (tnurlygayanov)]

Looks like we need to describe this issue in tempest and then commit the fix to tempest. We can mark this test as 'expected fail' while this issue not solved in Tempest.

[Timur Nurlygayanov (tnurlygayanov)]

This test case was added to should_fail file in this commit:
https://github.com/Mirantis/mos-tempest-runner/commit/d14ff1cb4a4d4e2af4bd2f3765f4b8a8f3b5afb0

[Timur Nurlygayanov (tnurlygayanov)]

We will describe separate bug in Tempest about this test case and we marked this test as 'should fail' because it doesn't work because of bug in Tempest tests.

[Timur Nurlygayanov (tnurlygayanov)]

We need to change status to Fix Released when this test will pass in MOS CI

[Timur Nurlygayanov (tnurlygayanov)]

This test was added to should fail file and should be skipped now.
This test will pass on environments with one controller and will fail on environments with 3 controllers because of neutron feature with multiple DHCP agents.