nova.tests.functional.test_server_group.ServerGroupTestV264.test_boot_servers_with_affinity_no_valid_host intermittently failing with "OpenStackApiNotFoundException: Item not found"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Matt Riedemann |
Bug Description
2018-10-29 10:14:46.550516 | ubuntu-xenial | 2018-10-29 10:14:45,457 INFO [nova.api.
2018-10-29 10:14:46.550841 | ubuntu-xenial | 2018-10-29 10:14:45,485 INFO [nova.api.
2018-10-29 10:14:46.551172 | ubuntu-xenial | 2018-10-29 10:14:45,639 INFO [nova.api.
2018-10-29 10:14:46.551333 | ubuntu-xenial | 2018-10-29 10:14:45,657 INFO [nova.filters] Filter ServerGroupAffi
2018-10-29 10:14:46.552003 | ubuntu-xenial | 2018-10-29 10:14:45,657 INFO [nova.filters] Filtering removed all hosts for the request with instance ID 'e5f8520d-
2018-10-29 10:14:46.552140 | ubuntu-xenial | 2018-10-29 10:14:45,658 ERROR [nova.conductor
2018-10-29 10:14:46.552209 | ubuntu-xenial | Traceback (most recent call last):
2018-10-29 10:14:46.552345 | ubuntu-xenial | File "nova/conductor
2018-10-29 10:14:46.552423 | ubuntu-xenial | instance_uuids, return_
2018-10-29 10:14:46.552545 | ubuntu-xenial | File "nova/conductor
2018-10-29 10:14:46.552626 | ubuntu-xenial | return_
2018-10-29 10:14:46.552740 | ubuntu-xenial | File "nova/scheduler
2018-10-29 10:14:46.552810 | ubuntu-xenial | return func(*args, **kwargs)
2018-10-29 10:14:46.552942 | ubuntu-xenial | File "nova/scheduler
2018-10-29 10:14:46.553051 | ubuntu-xenial | instance_uuids, return_objects, return_alternates)
2018-10-29 10:14:46.553167 | ubuntu-xenial | File "nova/scheduler
2018-10-29 10:14:46.553268 | ubuntu-xenial | return getattr(
2018-10-29 10:14:46.553387 | ubuntu-xenial | File "nova/scheduler
2018-10-29 10:14:46.553483 | ubuntu-xenial | instance_uuids, return_objects, return_alternates)
2018-10-29 10:14:46.553596 | ubuntu-xenial | File "nova/scheduler
2018-10-29 10:14:46.553703 | ubuntu-xenial | return cctxt.call(ctxt, 'select_
2018-10-29 10:14:46.553934 | ubuntu-xenial | File "/home/
2018-10-29 10:14:46.553995 | ubuntu-xenial | retry=self.retry)
2018-10-29 10:14:46.554235 | ubuntu-xenial | File "/home/
2018-10-29 10:14:46.554280 | ubuntu-xenial | retry=retry)
2018-10-29 10:14:46.554693 | ubuntu-xenial | File "/home/
2018-10-29 10:14:46.554906 | ubuntu-xenial | return self._send(target, ctxt, message, wait_for_reply, timeout)
2018-10-29 10:14:46.555228 | ubuntu-xenial | File "/home/
2018-10-29 10:14:46.555319 | ubuntu-xenial | raise failure
2018-10-29 10:14:46.555485 | ubuntu-xenial | NoValidHost: No valid host was found. There are not enough hosts available.
2018-10-29 10:14:46.555816 | ubuntu-xenial | 2018-10-29 10:14:45,697 WARNING [nova.scheduler
2018-10-29 10:14:46.556006 | ubuntu-xenial | 2018-10-29 10:14:45,698 WARNING [nova.scheduler
Looks like the test creates two servers in an affinity group and then tries to create a third server in the group and expects it to fail because there are not enough resources on the host with the other two servers.
That all seems to work. The problem is when creating the 3rd server, while waiting for its status to change to ERROR the API returns a 404:
2018-10-29 10:14:46.484852 | ubuntu-xenial | Captured traceback:
2018-10-29 10:14:46.484897 | ubuntu-xenial | ~~~~~~~~~~~~~~~~~~~
2018-10-29 10:14:46.484967 | ubuntu-xenial | Traceback (most recent call last):
2018-10-29 10:14:46.485159 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.485247 | ubuntu-xenial | expected_
2018-10-29 10:14:46.485394 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.485490 | ubuntu-xenial | self.admin_api, created_server, expected_status)
2018-10-29 10:14:46.485633 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.485743 | ubuntu-xenial | admin_api, server, {'status': expected_status}, max_retries)
2018-10-29 10:14:46.485892 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.485980 | ubuntu-xenial | server = admin_api.
2018-10-29 10:14:46.486104 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.486218 | ubuntu-xenial | return self.api_
2018-10-29 10:14:46.486330 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.486439 | ubuntu-xenial | return APIResponse(
2018-10-29 10:14:46.486557 | ubuntu-xenial | File "nova/tests/
2018-10-29 10:14:46.486671 | ubuntu-xenial | raise OpenStackApiNot
2018-10-29 10:14:46.486811 | ubuntu-xenial | nova.tests.
My guess would be something in this recent change is causing a problem looking up the instance from the cell0 DB:
http:// logstash. openstack. org/#/dashboard /file/logstash. json?query= message: %5C%22INFO% 20%5Bnova. filters% 5D%20Filter% 20ServerGroupAf finityFilter% 20returned% 200%20hosts% 5C%22%20AND% 20tags: %5C%22console% 5C%22&from= 10d