test_rebuild_server fails with mismatch - fixed and floating ip type after rebuild

[Martin Kopec]

master version of tempest, the following test fails with the following traceback in my deployment

tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server[id-aaa6cdf3-55a7-461a-add9-1c8596b9a07c]
----------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 244, in test_rebuild_server
    self._test_rebuild_server()

      File "/home/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 218, in _test_rebuild_server
    self.assertEqual(original_addresses, server['addresses'])

      File "/home/stack/tempest/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/stack/tempest/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = {'tempest-ServerActionsTestJSON-117661335-network': [{'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'fixed',
                                                      'addr': '10.100.0.14',
                                                      'version': 4}]}
actual = {'tempest-ServerActionsTestJSON-117661335-network': [{'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'fixed',
                                                      'addr': '10.100.0.14',
                                                      'version': 4},
                                                     {'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'floating',
                                                      'addr': '10.0.0.183',
                                                      'version': 4}]}

[Martin Kopec (mkopec)]

It seems that before the server had 2 ips (fixed and floating one) before the rebuild - at this point in the test:
https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187

[Martin Kopec (mkopec)]

There seems to be a race condition - it takes some time to get a floating ip assigned an the test doesn't take that into account:

I used a breakpoint and went step by step printing the variables, see below:

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187
>>> original_addresses
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10', 'OS-EXT-IPS:type': 'fixed',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}, {'version': 4, 'addr': '10.0.0.226', 'OS-EXT-IPS:type': 'floating',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}]}

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L193
>>> rebuilt_server['addresses']
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10'}, {'version': 4, 'addr':
'10.0.0.226'}]}

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L214
>>> server['addresses']
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10', 'OS-EXT-IPS:type': 'fixed',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}, {'version': 4, 'addr': '10.0.0.226', 'OS-EXT-IPS:type': 'floating',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}]}

rebuilt_server - server right after the rebuild didn't contain a floating ip, however a little bit later (line 214) it had a floating ip assigned already - there is a possibility that the ip doesn't get assigned between lines 193 and 214 and that will result in the mismatch

[Martin Kopec (mkopec)]

oh, it may happen also the other way around:
original_addresses here:
https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187
may not contain floating ip yet (if the server was built right before the test), however, the rebuild server may have it which will result in the mismatch again.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/814085

[Martin Kopec (mkopec)]

to conclude, floating_ip is still attached to a server even if the server is in a REBUILD state. So the only case we need to take care of is the situation when the test server is built right before the test which results in original_addresses not containing the floating ip.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/814085
Committed: https://opendev.org/openstack/tempest/commit/d4c9cc89f199899b20cfaea975ecd15ee66d3523
Submitter: "Zuul (22348)"
Branch: master

commit d4c9cc89f199899b20cfaea975ecd15ee66d3523
Author: Martin Kopec <email address hidden>
Date: Fri Oct 15 00:01:36 2021 +0000

    Fix test_rebuild_server test by waiting for floating ip

    The test could fail on an adresses mismatch when the test server
    didn't have floating ip assigned yet at the beginning of the test.

    Closes-Bug: #1945495
    Change-Id: Ic99ba61d87dfc24a684263d41e0caa4f54d1d263

[Martin Kopec (mkopec)]

The fix introduced couple of days ago had to be edited as it caused issues in neutron:
https://bugs.launchpad.net/neutron/+bug/1960022

The proper fix of this bug is:
https://review.opendev.org/c/openstack/tempest/+/828245

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 30.0.0 release.