TestServerMultinode.test_schedule_to_all_nodes fails in gate-tempest-dsvm-cells for Liberty

[Sylvain Bauza]

It seems that all gate-tempest-dsvm-cells runs against the liberty branch fail because of

2015-11-16 04:50:15.334 | ==============================
2015-11-16 04:50:15.334 | Failed 1 tests - output below:
2015-11-16 04:50:15.334 | ==============================
2015-11-16 04:50:15.334 |
2015-11-16 04:50:15.334 | tempest.scenario.test_server_multinode.TestServerMultinode.test_schedule_to_all_nodes[compute,id-9cecbe35-b9d4-48da-a37e-7ce70aa43d30,network,smoke]
2015-11-16 04:50:15.334 | ----------------------------------------------------------------------------------------------------------------------------------------------------
2015-11-16 04:50:15.335 |
2015-11-16 04:50:15.335 | Captured traceback:
2015-11-16 04:50:15.335 | ~~~~~~~~~~~~~~~~~~~
2015-11-16 04:50:15.335 | Traceback (most recent call last):
2015-11-16 04:50:15.335 | File "tempest/test.py", line 127, in wrapper
2015-11-16 04:50:15.335 | return f(self, *func_args, **func_kwargs)
2015-11-16 04:50:15.335 | File "tempest/scenario/test_server_multinode.py", line 75, in test_schedule_to_all_nodes
2015-11-16 04:50:15.335 | create_kwargs=create_kwargs)
2015-11-16 04:50:15.335 | File "tempest/scenario/manager.py", line 200, in create_server
2015-11-16 04:50:15.335 | status='ACTIVE')
2015-11-16 04:50:15.336 | File "tempest/common/waiters.py", line 75, in wait_for_server_status
2015-11-16 04:50:15.336 | server_id=server_id)
2015-11-16 04:50:15.336 | tempest.exceptions.BuildErrorException: Server 7b801bb9-c957-45cb-ab1e-0d802c9374c2 failed to build and is in ERROR status
2015-11-16 04:50:15.336 | Details: {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'details': u' File "/opt/stack/new/nova/nova/conductor/manager.py", line 739, in build_instances\n request_spec, filter_properties)\n File "/opt/stack/new/nova/nova/scheduler/utils.py", line 343, in wrapped\n return func(*args, **kwargs)\n File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 52, in select_destinations\n context, request_spec, filter_properties)\n File "/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in __run_method\n return getattr(self.instance, __name)(*args, **kwargs)\n File "/opt/stack/new/nova/nova/scheduler/client/query.py", line 34, in select_destinations\n context, request_spec, filter_properties)\n File "/opt/stack/new/nova/nova/scheduler/rpcapi.py", line 120, in select_destinations\n request_spec=request_spec, filter_properties=filter_properties)\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', u'created': u'2015-11-16T04:41:59Z'}
2015-11-16 04:50:15.336 |
2015-11-16 04:50:15.336 |
2015-11-16 04:50:15.336 | Captured traceback-1:
2015-11-16 04:50:15.336 | ~~~~~~~~~~~~~~~~~~~~~
2015-11-16 04:50:15.336 | Traceback (most recent call last):
2015-11-16 04:50:15.336 | File "tempest/common/waiters.py", line 111, in wait_for_server_termination
2015-11-16 04:50:15.336 | raise exceptions.BuildErrorException(server_id=server_id)
2015-11-16 04:50:15.337 | tempest.exceptions.BuildErrorException: Server 7b801bb9-c957-45cb-ab1e-0d802c9374c2 failed to build and is in ERROR status
2015-11-16 04:50:15.337 |
2015-11-16 04:50:15.337 |
2015-11-16 04:50:15.337 | Captured traceback-2:
2015-11-16 04:50:15.337 | ~~~~~~~~~~~~~~~~~~~~~
2015-11-16 04:50:15.337 | Traceback (most recent call last):
2015-11-16 04:50:15.337 | File "tempest/scenario/manager.py", line 143, in _wait_for_cleanups
2015-11-16 04:50:15.337 | waiter_callable(**wait)
2015-11-16 04:50:15.337 | File "tempest/common/waiters.py", line 111, in wait_for_server_termination
2015-11-16 04:50:15.337 | raise exceptions.BuildErrorException(server_id=server_id)
2015-11-16 04:50:15.337 | tempest.exceptions.BuildErrorException: Server 7b801bb9-c957-45cb-ab1e-0d802c9374c2 failed to build and is in ERROR status
2015-11-16 04:50:15.338 |

http://logs.openstack.org/20/243320/3/check/gate-tempest-dsvm-cells/5952387/console.html

[Sylvain Bauza (sylvain-bauza)]

So, the problem is due to http://logs.openstack.org/20/243320/3/check/gate-tempest-dsvm-cells/5952387/logs/screen-n-sch.txt.gz#_2015-11-16_04_41_58_981

It seems that since the cells API doesn't know the hosts correctly, using force_hosts is not working for Cells.

[Sylvain Bauza (sylvain-bauza)]

https://review.openstack.org/#/c/245382/ was merged in the master branch, we need to backport it to stable/liberty

[Augustina Ragwitz (auggy)]

Hi Sylvain, just checking in to see if this is still an issue.

[Sylvain Bauza (sylvain-bauza)]

Yeah the backport was merged too.

https://review.openstack.org/#/q/Icc71e36f4ecb015dff9e806caacd31262f7e17f7,n,z