test_rebuild_server fails with mismatch - fixed and floating ip type after rebuild

Bug #1945495 reported by Martin Kopec
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Fix Released
High
Martin Kopec

Bug Description

master version of tempest, the following test fails with the following traceback in my deployment

tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server[id-aaa6cdf3-55a7-461a-add9-1c8596b9a07c]
----------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 244, in test_rebuild_server
    self._test_rebuild_server()

      File "/home/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 218, in _test_rebuild_server
    self.assertEqual(original_addresses, server['addresses'])

      File "/home/stack/tempest/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/stack/tempest/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = {'tempest-ServerActionsTestJSON-117661335-network': [{'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'fixed',
                                                      'addr': '10.100.0.14',
                                                      'version': 4}]}
actual = {'tempest-ServerActionsTestJSON-117661335-network': [{'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'fixed',
                                                      'addr': '10.100.0.14',
                                                      'version': 4},
                                                     {'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a2:1f:b3',
                                                      'OS-EXT-IPS:type': 'floating',
                                                      'addr': '10.0.0.183',
                                                      'version': 4}]}

Martin Kopec (mkopec)
Changed in tempest:
assignee: nobody → Martin Kopec (mkopec)
Revision history for this message
Martin Kopec (mkopec) wrote :

It seems that before the server had 2 ips (fixed and floating one) before the rebuild - at this point in the test:
https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187

Revision history for this message
Martin Kopec (mkopec) wrote :

There seems to be a race condition - it takes some time to get a floating ip assigned an the test doesn't take that into account:

I used a breakpoint and went step by step printing the variables, see below:

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187
>>> original_addresses
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10', 'OS-EXT-IPS:type': 'fixed',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}, {'version': 4, 'addr': '10.0.0.226', 'OS-EXT-IPS:type': 'floating',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}]}

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L193
>>> rebuilt_server['addresses']
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10'}, {'version': 4, 'addr':
'10.0.0.226'}]}

https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L214
>>> server['addresses']
{'tempest-ServerActionsTestJSON-1379249512-network': [{'version': 4, 'addr': '10.100.0.10', 'OS-EXT-IPS:type': 'fixed',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}, {'version': 4, 'addr': '10.0.0.226', 'OS-EXT-IPS:type': 'floating',
'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:14:09:08'}]}

rebuilt_server - server right after the rebuild didn't contain a floating ip, however a little bit later (line 214) it had a floating ip assigned already - there is a possibility that the ip doesn't get assigned between lines 193 and 214 and that will result in the mismatch

Martin Kopec (mkopec)
Changed in tempest:
status: New → In Progress
importance: Undecided → High
Revision history for this message
Martin Kopec (mkopec) wrote :

oh, it may happen also the other way around:
original_addresses here:
https://opendev.org/openstack/tempest/src/commit/669900f622ce46c62260910d2cf9fa1d32216191/tempest/api/compute/servers/test_server_actions.py#L187
may not contain floating ip yet (if the server was built right before the test), however, the rebuild server may have it which will result in the mismatch again.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/814085

Revision history for this message
Martin Kopec (mkopec) wrote :

to conclude, floating_ip is still attached to a server even if the server is in a REBUILD state. So the only case we need to take care of is the situation when the test server is built right before the test which results in original_addresses not containing the floating ip.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.opendev.org/c/openstack/tempest/+/814085
Committed: https://opendev.org/openstack/tempest/commit/d4c9cc89f199899b20cfaea975ecd15ee66d3523
Submitter: "Zuul (22348)"
Branch: master

commit d4c9cc89f199899b20cfaea975ecd15ee66d3523
Author: Martin Kopec <email address hidden>
Date: Fri Oct 15 00:01:36 2021 +0000

    Fix test_rebuild_server test by waiting for floating ip

    The test could fail on an adresses mismatch when the test server
    didn't have floating ip assigned yet at the beginning of the test.

    Closes-Bug: #1945495
    Change-Id: Ic99ba61d87dfc24a684263d41e0caa4f54d1d263

Changed in tempest:
status: In Progress → Fix Released
Revision history for this message
Martin Kopec (mkopec) wrote :

The fix introduced couple of days ago had to be edited as it caused issues in neutron:
https://bugs.launchpad.net/neutron/+bug/1960022

The proper fix of this bug is:
https://review.opendev.org/c/openstack/tempest/+/828245

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tempest 30.0.0

This issue was fixed in the openstack/tempest 30.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.