test_create_server_with_pci_dev_and_numa functional test randomly fails with "OpenStackApiNotFoundException: Item not found"

[Matt Riedemann]

Seen here:

http://logs.openstack.org/03/448203/2/check/gate-nova-tox-functional-py35-ubuntu-xenial/5f60ae1/console.html#_2017-03-21_18_29_51_237243

2017-03-21 18:29:51.237243 | nova.tests.functional.libvirt.test_pci_sriov_servers.SRIOVServersTest.test_create_server_with_pci_dev_and_numa
2017-03-21 18:29:51.237307 | --------------------------------------------------------------------------------------------------------------
2017-03-21 18:29:51.237322 |
2017-03-21 18:29:51.237340 | Captured traceback:
2017-03-21 18:29:51.237358 | ~~~~~~~~~~~~~~~~~~~
2017-03-21 18:29:51.237381 | b'Traceback (most recent call last):'
2017-03-21 18:29:51.237440 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/.tox/functional-py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched'
2017-03-21 18:29:51.237466 | b' return func(*args, **keywargs)'
2017-03-21 18:29:51.237530 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 272, in test_create_server_with_pci_dev_and_numa'
2017-03-21 18:29:51.237572 | b' pf_server = self._run_build_test(flavor_id, filter_mock)'
2017-03-21 18:29:51.237632 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/libvirt/test_pci_sriov_servers.py", line 124, in _run_build_test'
2017-03-21 18:29:51.237666 | b" found_server = self._wait_for_state_change(found_server, 'BUILD')"
2017-03-21 18:29:51.237731 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/test_servers.py", line 64, in _wait_for_state_change'
2017-03-21 18:29:51.237782 | b" server = self.api.get_server(server['id'])"
2017-03-21 18:29:51.237836 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 250, in get_server'
2017-03-21 18:29:51.237870 | b" return self.api_get('/servers/%s' % server_id).body['server']"
2017-03-21 18:29:51.237919 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 207, in api_get'
2017-03-21 18:29:51.237951 | b' return APIResponse(self.api_request(relative_uri, **kwargs))'
2017-03-21 18:29:51.238002 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 189, in api_request'
2017-03-21 18:29:51.238032 | b' raise OpenStackApiNotFoundException(response=response)'
2017-03-21 18:29:51.238067 | b'nova.tests.functional.api.client.OpenStackApiNotFoundException: Item not found'
2017-03-21 18:29:51.238085 | b''

Looks like it fails while waiting for a server to be active.

The test is polling every .1 second, so I'm wondering if we're hitting a window between the time that the instance is created and mapped to a cell and the build_request is deleted, but I thought we hit something like that in Tempest in Ocata and fixed it (but Tempest polls every 1 second, not every .1 second).

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%5C%22nova.tests.functional.api.client.OpenStackApiNotFoundException%3A%20Item%20not%20found%5C%22%20OR%20message%3A%5C%22b'nova.tests.functional.api.client.OpenStackApiNotFoundException%3A%20Item%20not%20found'%5C%22)%20AND%20tags%3A%5C%22console%5C%22&from=7d

189 hits in 7 days, check queue only but multiple changes.

[Matt Riedemann (mriedem)]

The test is waiting for the server to go to from BUILD status and then once it does, it asserts that it's status is ACTIVE.

We have other functional tests that just wait for the server to go to ACTIVE status or timeout. That's duplicated in a lot of functional tests. We should probably just have some wait utilities for the functional tests to use so we don't have to duplicate this logic all over the tests.

[Alex Xu (xuhj)]

Similar one show up at here http://logs.openstack.org/32/461732/2/check/gate-nova-tox-functional-ubuntu-xenial/83038b5/testr_results.html.gz

[Sean Dague (sdague)]

Automatically discovered version ocata in description. If this is incorrect, please update the description to include 'nova version: ...'