kolla-ansible

No valid host was found error

Bug #1723155 reported by José Donoso on 2017-10-12

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Expired	Undecided	Unassigned

Bug Description

Description
===========
When i try to launch an instance the following error appear: "No valid host was found"

Expected result
===============
Instance succesfully launched

Actual result
=============
Failed to perform requested operation on instance "c", the instance has an error status: Please try again later [Error: No valid host was found. ]

Environment
===========
I deployed Kolla-Ansible Openstack from binary with yum so is the latest version.
I have one controller network monitoring and compute node, one compute-only node and one storage-only node.
I have two interfaces in every node, one is the kolla internal and external interface and one is the neutron external interface

Im using kvm as hypervisor
I installed libvirtd service in the controller node after the deployment and is the only node with libvirtd installed

Ceph as backend

Logs & Configs
==============
In nova-api.log

2017-10-12 12:14:00.942 25 DEBUG nova.osapi_compute.wsgi.server [req-10b4d056-6933-4fa6-ba9f-162bc65945fd - - - - -] (25) accepted ('172.30.220.3', 55626) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-12 12:14:00.956 25 DEBUG nova.api.openstack.wsgi [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Action: 'create', calling method: <bound method ServersController.create of <nova.api.openstack.compute.servers.ServersController object at 0x848bad0>>, body: {"server": {"name": "c", "imageRef": "", "availability_zone": "nova", "key_name": "mykey", "flavorRef": "3", "OS-DCF:diskConfig": "AUTO", "max_count": 1, "block_device_mapping_v2": [{"boot_index": "0", "uuid": "e8e6918b-20cc-4d79-905a-6556f6b0e35c", "volume_size": 40, "device_name": "vda", "source_type": "image", "destination_type": "volume", "delete_on_termination": false}], "min_count": 1, "networks": [{"uuid": "c539fd11-f789-43d0-81f1-d3809746aa8e"}], "security_groups": [{"name": "9203d5cb-7ee5-4f34-a600-867e29b1a48e"}]}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:609
2017-10-12 12:14:01.014 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.014 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.015 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.016 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.037 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.038 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.058 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.058 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.509 25 DEBUG nova.network.neutronv2.api [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] validate_networks() for [(u'c539fd11-f789-43d0-81f1-d3809746aa8e', None, None, None)] validate_networks /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1679
2017-10-12 12:14:01.706 25 DEBUG nova.virt.hardware [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] emulator threads policy constraint: None get_emulator_threads_constraint /usr/lib/python2.7/site-packages/nova/virt/hardware.py:1287
2017-10-12 12:14:01.718 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for project 189671b052b34af8b34edbf9a3f87e76. Resources: ['metadata_items'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:442
2017-10-12 12:14:01.723 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for user d45a3d94c6fc4e34b7b9d6bbab8277b4 and project 189671b052b34af8b34edbf9a3f87e76. Resources: ['metadata_items'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:434
2017-10-12 12:14:01.745 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for project 189671b052b34af8b34edbf9a3f87e76. Resources: ['injected_files'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:442
2017-10-12 12:14:01.749 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for user d45a3d94c6fc4e34b7b9d6bbab8277b4 and project 189671b052b34af8b34edbf9a3f87e76. Resources: ['injected_files'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:434
2017-10-12 12:14:01.769 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for project 189671b052b34af8b34edbf9a3f87e76. Resources: ['injected_file_content_bytes', 'injected_file_path_bytes'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:442
2017-10-12 12:14:01.773 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for user d45a3d94c6fc4e34b7b9d6bbab8277b4 and project 189671b052b34af8b34edbf9a3f87e76. Resources: ['injected_file_content_bytes', 'injected_file_path_bytes'] _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:434
2017-10-12 12:14:01.785 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.786 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.786 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:01.787 25 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:01.806 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for project 189671b052b34af8b34edbf9a3f87e76. Resources: set(['instances', 'ram', 'cores']) _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:442
2017-10-12 12:14:01.811 25 DEBUG nova.quota [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Getting quotas for user d45a3d94c6fc4e34b7b9d6bbab8277b4 and project 189671b052b34af8b34edbf9a3f87e76. Resources: set(['instances', 'ram', 'cores']) _get_quotas /usr/lib/python2.7/site-packages/nova/quota.py:434
2017-10-12 12:14:01.824 25 DEBUG nova.compute.api [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Going to run 1 instances... _provision_instances /usr/lib/python2.7/site-packages/nova/compute/api.py:891
2017-10-12 12:14:01.839 25 DEBUG nova.compute.api [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] [instance: 0e33fd82-a847-4b77-8290-01af48faf79f] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=False,deleted=<?>,deleted_at=<?>,destination_type='volume',device_name='/dev/vda',device_type=None,disk_bus=None,guest_format=None,id=<?>,image_id='e8e6918b-20cc-4d79-905a-6556f6b0e35c',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,volume_id=None,volume_size=40)] _bdm_validate_set_size_and_instance /usr/lib/python2.7/site-packages/nova/compute/api.py:1268
2017-10-12 12:14:01.927 25 INFO nova.osapi_compute.wsgi.server [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "POST /v2.1/189671b052b34af8b34edbf9a3f87e76/os-volumes_boot HTTP/1.1" status: 202 len: 975 time: 0.9833241
2017-10-12 12:14:01.949 25 DEBUG nova.api.openstack.wsgi [req-5d32880e-5b39-40bb-8194-e8c583a45b47 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x886b6d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:01.957 25 DEBUG nova.compute.api [req-5d32880e-5b39-40bb-8194-e8c583a45b47 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] [instance: 0e33fd82-a847-4b77-8290-01af48faf79f] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2279
2017-10-12 12:14:02.023 25 DEBUG nova.policy [req-5d32880e-5b39-40bb-8194-e8c583a45b47 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'service_roles': [], 'user_id': u'd45a3d94c6fc4e34b7b9d6bbab8277b4', 'roles': [u'heat_stack_owner', u'admin'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': True, 'project_id': u'189671b052b34af8b34edbf9a3f87e76', 'project_domain_id': u'default'} authorize /usr/lib/python2.7/site-packages/nova/policy.py:168
2017-10-12 12:14:02.080 25 INFO nova.osapi_compute.wsgi.server [req-5d32880e-5b39-40bb-8194-e8c583a45b47 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/servers/0e33fd82-a847-4b77-8290-01af48faf79f HTTP/1.1" status: 200 len: 1599 time: 0.1470160
2017-10-12 12:14:02.170 25 DEBUG nova.osapi_compute.wsgi.server [req-10b4d056-6933-4fa6-ba9f-162bc65945fd - - - - -] (25) accepted ('172.30.220.3', 55674) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-12 12:14:02.170 24 DEBUG nova.osapi_compute.wsgi.server [req-a8cc65c5-82c2-42a1-92e5-2b913e946d96 - - - - -] (24) accepted ('172.30.220.3', 55678) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-12 12:14:02.182 24 DEBUG nova.api.openstack.wsgi [req-762e9cba-6142-42a2-bfad-7738c8aaeb91 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method FlavorsController.detail of <nova.api.openstack.compute.flavors.FlavorsController object at 0x8146890>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:02.185 25 INFO nova.osapi_compute.wsgi.server [req-a2a8d9a8-21fa-46c3-9457-aee1e12b5aaa d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76 HTTP/1.1" status: 404 len: 358 time: 0.0133569
2017-10-12 12:14:02.199 25 DEBUG nova.api.openstack.wsgi [req-ae24dfb9-e8ef-4d52-b858-4d8575bca1c6 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x8146050>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:02.201 25 INFO nova.osapi_compute.wsgi.server [req-ae24dfb9-e8ef-4d52-b858-4d8575bca1c6 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/ HTTP/1.1" status: 200 len: 763 time: 0.0105250
2017-10-12 12:14:02.230 24 INFO nova.osapi_compute.wsgi.server [req-762e9cba-6142-42a2-bfad-7738c8aaeb91 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/flavors/detail HTTP/1.1" status: 200 len: 2495 time: 0.0585120
2017-10-12 12:14:02.285 24 DEBUG nova.osapi_compute.wsgi.server [req-a8cc65c5-82c2-42a1-92e5-2b913e946d96 - - - - -] (24) accepted ('172.30.220.3', 55696) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-12 12:14:02.363 24 DEBUG nova.api.openstack.wsgi [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x872c390>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:02.371 24 DEBUG nova.compute.api [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Searching by: {'deleted': False, u'project_id': u'189671b052b34af8b34edbf9a3f87e76'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2311
2017-10-12 12:14:02.389 24 DEBUG oslo_concurrency.lockutils [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:02.390 24 DEBUG oslo_concurrency.lockutils [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:02.438 24 DEBUG nova.compute.api [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Skipping already-collected cell0 list _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2503
2017-10-12 12:14:02.439 24 DEBUG nova.compute.api [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Listing 20 instances in cell e4e34c46-c1ef-4480-84a6-9dd3996a0a82 _get_instances_by_filters_all_cells /usr/lib/python2.7/site-packages/nova/compute/api.py:2506
2017-10-12 12:14:02.439 24 DEBUG oslo_concurrency.lockutils [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:02.440 24 DEBUG oslo_concurrency.lockutils [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:02.508 24 DEBUG nova.policy [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'service_roles': [], 'user_id': u'd45a3d94c6fc4e34b7b9d6bbab8277b4', 'roles': [u'heat_stack_owner', u'admin'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': True, 'project_id': u'189671b052b34af8b34edbf9a3f87e76', 'project_domain_id': u'default'} authorize /usr/lib/python2.7/site-packages/nova/policy.py:168
2017-10-12 12:14:02.666 24 INFO nova.osapi_compute.wsgi.server [req-f9b96e2b-4b07-4d4e-8415-21c6f51b51bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/servers/detail?limit=21&project_id=189671b052b34af8b34edbf9a3f87e76 HTTP/1.1" status: 200 len: 1932 time: 0.3798020
2017-10-12 12:14:03.362 24 DEBUG nova.api.openstack.wsgi [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<function version_select at 0x9f00050>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:03.400 24 DEBUG oslo_concurrency.lockutils [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:03.401 24 DEBUG oslo_concurrency.lockutils [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:03.402 24 DEBUG oslo_concurrency.lockutils [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:03.402 24 DEBUG oslo_concurrency.lockutils [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:03.435 24 INFO nova.osapi_compute.wsgi.server [req-d392ac5e-7c85-4ab6-b4ed-f19064747e71 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/limits?reserved=1 HTTP/1.1" status: 200 len: 889 time: 0.0831459
2017-10-12 12:14:03.533 24 INFO nova.osapi_compute.wsgi.server [req-63d4e421-fafa-499c-ba22-364a387e26e0 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76 HTTP/1.1" status: 404 len: 358 time: 0.0092881
2017-10-12 12:14:03.544 24 DEBUG nova.api.openstack.wsgi [req-bdd4a3f5-e098-462b-a4e7-bff2926b0dfc d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x8146050>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:03.549 24 INFO nova.osapi_compute.wsgi.server [req-bdd4a3f5-e098-462b-a4e7-bff2926b0dfc d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/ HTTP/1.1" status: 200 len: 763 time: 0.0131600
2017-10-12 12:14:03.602 24 DEBUG nova.api.openstack.wsgi [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<function version_select at 0x9eda5f0>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:03.635 24 DEBUG oslo_concurrency.lockutils [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:03.635 24 DEBUG oslo_concurrency.lockutils [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:03.636 24 DEBUG oslo_concurrency.lockutils [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:03.637 24 DEBUG oslo_concurrency.lockutils [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "e4e34c46-c1ef-4480-84a6-9dd3996a0a82" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:03.666 24 INFO nova.osapi_compute.wsgi.server [req-9107cb8a-2741-4e1b-99aa-efe4c56ed8bb d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/limits?reserved=1 HTTP/1.1" status: 200 len: 889 time: 0.0745518
2017-10-12 12:14:04.702 24 INFO nova.osapi_compute.wsgi.server [req-4a792988-e9f8-4a33-a8c7-4c09f0dacbd5 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76 HTTP/1.1" status: 404 len: 358 time: 0.0117171
2017-10-12 12:14:04.716 24 DEBUG nova.api.openstack.wsgi [req-d76f4ea3-ab01-4b6b-8581-ab64bbb6dfcd d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x8146050>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:04.718 24 INFO nova.osapi_compute.wsgi.server [req-d76f4ea3-ab01-4b6b-8581-ab64bbb6dfcd d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/ HTTP/1.1" status: 200 len: 763 time: 0.0107520
2017-10-12 12:14:04.811 26 DEBUG nova.osapi_compute.wsgi.server [req-3f2dff86-6e4f-4423-ae35-a3d0c93b5a02 - - - - -] (26) accepted ('172.30.220.3', 55820) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-12 12:14:04.905 26 DEBUG nova.api.openstack.wsgi [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x886b6d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:04.910 26 DEBUG nova.compute.api [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] [instance: 0e33fd82-a847-4b77-8290-01af48faf79f] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2279
2017-10-12 12:14:04.921 26 DEBUG oslo_concurrency.lockutils [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:04.921 26 DEBUG oslo_concurrency.lockutils [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:04.996 26 DEBUG oslo_concurrency.lockutils [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:04.996 26 DEBUG oslo_concurrency.lockutils [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:04.996 26 DEBUG nova.objects.instance [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lazy-loading 'fault' on Instance uuid 0e33fd82-a847-4b77-8290-01af48faf79f obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:1048
2017-10-12 12:14:05.022 26 DEBUG nova.policy [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'service_roles': [], 'user_id': u'd45a3d94c6fc4e34b7b9d6bbab8277b4', 'roles': [u'heat_stack_owner', u'admin'], 'user_domain_id': u'default', 'service_project_id': None, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_domain_id': None, 'is_admin_project': True, 'is_admin': True, 'project_id': u'189671b052b34af8b34edbf9a3f87e76', 'project_domain_id': u'default'} authorize /usr/lib/python2.7/site-packages/nova/policy.py:168
2017-10-12 12:14:05.166 26 INFO nova.osapi_compute.wsgi.server [req-55f99f98-edd1-43ff-8a05-476f3ca6abbf d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/servers/0e33fd82-a847-4b77-8290-01af48faf79f HTTP/1.1" status: 200 len: 3516 time: 0.3529608
2017-10-12 12:14:05.184 24 DEBUG nova.api.openstack.wsgi [req-63707545-e8c8-4e03-a55a-fa0776d39f11 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method FlavorsController.show of <nova.api.openstack.compute.flavors.FlavorsController object at 0x8146dd0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:05.204 24 INFO nova.osapi_compute.wsgi.server [req-63707545-e8c8-4e03-a55a-fa0776d39f11 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76/flavors/3 HTTP/1.1" status: 200 len: 807 time: 0.0318151
2017-10-12 12:14:05.632 24 INFO nova.osapi_compute.wsgi.server [req-d879c0bb-c59a-4820-a300-09ccb7eac5a4 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/189671b052b34af8b34edbf9a3f87e76 HTTP/1.1" status: 404 len: 358 time: 0.0108509
2017-10-12 12:14:05.644 24 DEBUG nova.api.openstack.wsgi [req-b6c86537-d323-41b6-b118-fc1ca8ebf8a5 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x8146050>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-10-12 12:14:05.645 24 INFO nova.osapi_compute.wsgi.server [req-b6c86537-d323-41b6-b118-fc1ca8ebf8a5 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] 172.30.220.254,172.30.220.3 "GET /v2.1/ HTTP/1.1" status: 200 len: 763 time: 0.0101080

And in nova-conductor.log

2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 137, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1027, in schedule_and_build_instances
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager instance_uuids)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 626, in _schedule_instances
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager request_spec, instance_uuids)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 586, in wrapped
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 52, in select_destinations
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager instance_uuids)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 33, in select_destinations
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager instance_uuids)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 137, in select_destinations
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager retry=self.retry)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager retry=retry)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager raise result
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 137, in select_destinations
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager
2017-10-12 12:14:02.686 22 ERROR nova.conductor.manager
2017-10-12 12:14:02.697 22 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:02.698 22 DEBUG oslo_concurrency.lockutils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:02.743 22 WARNING nova.scheduler.utils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Failed to compute_task_build_instances: No valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 137, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
: NoValidHost_Remote: No valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 137, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.
2017-10-12 12:14:02.744 22 WARNING nova.scheduler.utils [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] [instance: 0e33fd82-a847-4b77-8290-01af48faf79f] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 232, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 137, in select_destinations
raise exception.NoValidHost(reason="")

NoValidHost: No valid host was found.

Inventory file

# These initial groups are the only groups required to be modified. The
# additional groups are for more control of the environment.
[control]
# These hostname must be resolvable from your deployment host
172.30.220.3

# The above can also be specified as follows:
#control[01:03] ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
172.30.220.3

[compute]
172.30.220.3
172.30.220.4

[monitoring]
172.30.220.3

# When compute nodes and control nodes use different interfaces,
# you can specify "api_interface" and other interfaces like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]
172.30.220.5

[deployment]
localhost ansible_connection=local

[targets]
172.30.220.3
172.30.220.4
172.30.220.5

[targets:vars]
ansible_user=*******
ansible_ssh_pass=******

the rest is the same as the multinode default file

See original description

Tags:

José Donoso (jose.manuel.akainix) on 2017-10-12

description:

updated

Revision history for this message

Eduardo Gonzalez (egonzalez90) wrote on 2017-10-12:

Please share nova compute logs in compute host.

Changed in kolla-ansible:
status:	New → Incomplete

Revision history for this message

Eduardo Gonzalez (egonzalez90) wrote on 2017-10-12:

Also, share scheduler logs, globals.yml config, and inventory file(change IPs to different one if needed)

Revision history for this message

José Donoso (jose.manuel.akainix) wrote on 2017-10-12:

Download full text (15.3 KiB)

scheduler.log

2017-10-12 12:14:01.967 7 DEBUG nova.scheduler.manager [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Starting to schedule for instances: [u'0e33fd82-a847-4b77-8290-01af48faf79f'] select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/manager.py:113
2017-10-12 12:14:02.680 7 DEBUG nova.scheduler.manager [req-efa73f8e-c7c6-4e0e-bf1a-712b7d5afa29 d45a3d94c6fc4e34b7b9d6bbab8277b4 189671b052b34af8b34edbf9a3f87e76 - default default] Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service. select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/manager.py:133
2017-10-12 12:14:03.754 7 DEBUG oslo_concurrency.lockutils [req-a2d9faea-d259-4315-a3fc-31cfafaaed4c - - - - -] Lock "host_instance" acquired by "nova.scheduler.host_manager.sync_instance_info" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-10-12 12:14:03.755 7 INFO nova.scheduler.host_manager [req-a2d9faea-d259-4315-a3fc-31cfafaaed4c - - - - -] Successfully synced instances from host 'chihiro.akainix.local'.
2017-10-12 12:14:03.755 7 DEBUG oslo_concurrency.lockutils [req-a2d9faea-d259-4315-a3fc-31cfafaaed4c - - - - -] Lock "host_instance" released by "nova.scheduler.host_manager.sync_instance_info" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-10-12 12:14:04.057 7 DEBUG oslo_service.periodic_task [req-9d92bc3f-18a0-4481-8fc3-218fe991a94b - - - - -] Running periodic task SchedulerManager._run_periodic_tasks run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215

Inventory file

# The above can also be specified as follows:
#control[01:03] ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
172.30.220.3

[compute]
172.30.220.3
172.30.220.4

[monitoring]
172.30.220.3

[storage]
172.30.220.5

[deployment]
localhost ansible_connection=local

[targets]
172.30.220.3
172.30.220.4
172.30.220.5

[targets:vars]
ansible_user=*******
ansible_ssh_pass=******

the rest is the same as the multinode default file

And with the globals.yml file

---
# You can use this file to override _any_ variable throughout Kolla.
# Additional options can be found in the
# 'kolla-ansible/ansible/group_vars/all.yml' file. Default value of all the
# commented parameters are shown here, To override the default value uncomment
# the parameter and change its value.

###############
# Kolla options
###############
# Va...

scheduler.log

Inventory file

# The above can also be specified as follows:
#control[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
172.30.220.3

[compute]
172.30.220.3
172.30.220.4

[monitoring]
172.30.220.3

[storage]
172.30.220.5

[deployment]
localhost       ansible_connection=local

[targets]
172.30.220.3
172.30.220.4
172.30.220.5

[targets:vars]
ansible_user=*******
ansible_ssh_pass=******

the rest is the same as the multinode default file

And with the globals.yml file

###############
# Kolla options
###############
# Valid options are [ COPY_ONCE, COPY_ALWAYS ]
#config_strategy: "COPY_ALWAYS"

# Valid options are [ centos, oraclelinux, ubuntu ]
#kolla_base_distro: "centos"

# Valid options are [ binary, source ]
#kolla_install_type: "binary"

# Valid option is Docker repository tag
#openstack_release: ""

# Location of configuration overrides
#node_custom_config: "/etc/kolla/config"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. If you want to run an
# All-In-One without haproxy and keepalived, you can set enable_haproxy to no
# in "OpenStack options" section, and set this value to the first IP on your
# 'network_interface' as set in the Networking section below.
kolla_internal_vip_address: "172.30.220.254"

# This is the DNS name that maps to the kolla_internal_vip_address VIP. By
# default it is the same as kolla_internal_vip_address.
#kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability. It defaults to the
# kolla_internal_vip_address, allowing internal and external communication to
# share the same address.  Specify a kolla_external_vip_address to separate
# internal and external requests between two VIPs.
#kolla_external_vip_address: "{{ kolla_internal_vip_address }}"
#kolla_external_vip_address: "172.30.220.254"

# The Public address used to communicate with OpenStack as set in the public_url
# for the endpoints that will be created. This DNS name should map to
# kolla_external_vip_address.
#kolla_external_fqdn: "{{ kolla_external_vip_address }}"

################
# Docker options
################
# Below is an example of a private repository with authentication. Note the
# Docker registry password can also be set in the passwords.yml file.

docker_registry: "172.30.220.2:5000"
#docker_namespace: "companyname"
#docker_registry_username: "sam"
#docker_registry_password: "correcthorsebatterystaple"

##############################
# Neutron - Networking Options
##############################
# This interface is what all your api services will be bound to by default.
# Additionally, all vxlan/tunnel and storage network traffic will go over this
# interface by default. This interface must contain an IPv4 address.
# It is possible for hosts to have non-matching names of interfaces - these can
# be set in an inventory file per host or per group or stored separately, see
#     http://docs.ansible.com/ansible/intro_inventory.html
# Yet another way to workaround the naming problem is to create a bond for the
# interface on all hosts and give the bond name here. Similar strategy can be
# followed for other types of interfaces.
network_interface: "em1"

# These can be adjusted for even more customization. The default is the same as
# the 'network_interface'. These interfaces must contain an IPv4 address.
#kolla_external_vip_interface: "{{ network_interface }}"
#kolla_external_vip_interface: "em1"
#api_interface: "{{ network_interface }}"
#storage_interface: "{{ network_interface }}"
#cluster_interface: "{{ network_interface }}"
#tunnel_interface: "{{ network_interface }}"
#dns_interface: "{{ network_interface }}"

# This is the raw interface given to neutron as its external network port. Even
# though an IP address can exist on this interface, it will be unusable in most
# configurations. It is recommended this interface not be configured with any IP
# addresses for that reason.
neutron_external_interface: "em2"

# Valid options are [ openvswitch, linuxbridge, vmware_nsxv, vmware_dvs, opendaylight ]
#neutron_plugin_agent: "openvswitch"

####################
# keepalived options
####################
# Arbitrary unique number from 0..255
#keepalived_virtual_router_id: "51"

#############
# TLS options
#############
# To provide encryption and authentication on the kolla_external_vip_interface,
# TLS can be enabled.  When TLS is enabled, certificates must be provided to
# allow clients to perform authentication.
#kolla_enable_tls_external: "no"
#kolla_external_fqdn_cert: "{{ node_config_directory }}/certificates/haproxy.pem"

##############
# OpenDaylight
##############
#enable_opendaylight_qos: "no"
#enable_opendaylight_l3: "yes"

###################
# OpenStack options
###################
# Use these options to set the various log levels across all OpenStack projects
# Valid options are [ True, False ]
openstack_logging_debug: "True"

# Valid options are [ novnc, spice ]
#nova_console: "novnc"

# OpenStack services can be enabled or disabled with these options
#enable_aodh: "no"
#enable_barbican: "no"
#enable_ceilometer: "no"
#enable_central_logging: "no"
enable_ceph: "yes"
enable_ceph_rgw: "yes"
#enable_chrony: "no"
enable_cinder: "yes"
enable_cinder_backup: "yes"
#enable_cinder_backend_hnas_iscsi: "no"
#enable_cinder_backend_hnas_nfs: "no"
#enable_cinder_backend_iscsi: "no"
#enable_cinder_backend_lvm: "no"
#enable_cinder_backend_nfs: "no"
#enable_cloudkitty: "no"
#enable_collectd: "no"
#enable_congress: "no"
#enable_designate: "no"
#enable_destroy_images: "no"
#enable_etcd: "no"
#enable_freezer: "no"
#enable_gnocchi: "no"
enable_grafana: "no"
enable_haproxy: "yes"
enable_heat: "yes"
enable_horizon: "yes"
#enable_horizon_cloudkitty: "{{ enable_cloudkitty | bool }}"
#enable_horizon_designate: "{{ enable_designate | bool }}"
#enable_horizon_freezer: "{{ enable_freezer | bool }}"
#enable_horizon_ironic: "{{ enable_ironic | bool }}"
#enable_horizon_karbor: "{{ enable_karbor | bool }}"
#enable_horizon_magnum: "{{ enable_magnum | bool }}"
#enable_horizon_manila: "{{ enable_manila | bool }}"
#enable_horizon_mistral: "{{ enable_mistral | bool }}"
#enable_horizon_murano: "{{ enable_murano | bool }}"
#enable_horizon_neutron_lbaas: "{{ enable_neutron_lbaas | bool }}"
#enable_horizon_sahara: "{{ enable_sahara | bool }}"
#enable_horizon_searchlight: "{{ enable_searchlight | bool }}"
#enable_horizon_senlin: "{{ enable_senlin | bool }}"
#enable_horizon_solum: "{{ enable_solum | bool }}"
#enable_horizon_tacker: "{{ enable_tacker | bool }}"
#enable_horizon_trove: "{{ enable_trove | bool }}"
#enable_horizon_watcher: "{{ enable_watcher | bool }}"
#enable_horizon_zun: "{{ enable_zun | bool }}"
#enable_hyperv: "no"
#enable_influxdb: "no"
#enable_ironic: "no"
#enable_karbor: "no"
#enable_kuryr: "no"
#enable_magnum: "no"
#enable_manila: "no"
#enable_manila_backend_generic: "no"
#enable_manila_backend_hnas: "no"
#enable_mistral: "no"
#enable_mongodb: "no"
#enable_murano: "no"
#enable_multipathd: "no"
#enable_neutron_bgp_dragent: "no"
#enable_neutron_dvr: "no"
#enable_neutron_lbaas: "no"
#enable_neutron_fwaas: "no"
#enable_neutron_qos: "no"
#enable_neutron_agent_ha: "no"
#enable_neutron_vpnaas: "no"
#enable_neutron_sfc: "no"
#enable_nova_serialconsole_proxy: "no"
#enable_octavia: "no"
#enable_opendaylight: "no"
#enable_openvswitch: "{{ neutron_plugin_agent != 'linuxbridge' }}"
#enable_ovs_dpdk: "no"
#enable_osprofiler: "no"
#enable_panko: "no"
#enable_qdrouterd: "no"
#enable_rally: "no"
#enable_redis: "no"
#enable_sahara: "no"
#enable_searchlight: "no"
#enable_senlin: "no"
#enable_skydive: "no"
#enable_solum: "no"
#enable_swift: "no"
#enable_telegraf: "no"
#enable_tacker: "no"
#enable_tempest: "no"
#enable_trove: "no"
#enable_vmtp: "no"
#enable_watcher: "no"
#enable_zun: "no"

##############
# Ceph options
##############
# Ceph can be setup with a caching to improve performance. To use the cache you
# must provide separate disks than those for the OSDs
#ceph_enable_cache: "no"

# Set to no if using external Ceph without cephx.
#external_ceph_cephx_enabled: "yes"

# Ceph is not able to determine the size of a cache pool automatically,
# so the configuration on the absolute size is required here, otherwise the flush/evict will not work.
#ceph_target_max_bytes: ""
#ceph_target_max_objects: ""

# Valid options are [ forward, none, writeback ]
#ceph_cache_mode: "writeback"

# A requirement for using the erasure-coded pools is you must setup a cache tier
# Valid options are [ erasure, replicated ]
#ceph_pool_type: "replicated"

# Integrate ceph rados object gateway with openstack keystone
enable_ceph_rgw_keystone: "yes"

#############################
# Keystone - Identity Options
#############################

# Valid options are [ uuid, fernet ]
#keystone_token_provider: 'uuid'

# Interval to rotate fernet keys by (in seconds). Must be an interval of
# 60(1 min), 120(2 min), 180(3 min), 240(4 min), 300(5 min), 360(6 min),
# 600(10 min), 720(12 min), 900(15 min), 1200(20 min), 1800(30 min),
# 3600(1 hour), 7200(2 hour), 10800(3 hour), 14400(4 hour), 21600(6 hour),
# 28800(8 hour), 43200(12 hour), 86400(1 day), 604800(1 week).
#fernet_token_expiry: 86400

########################
# Glance - Image Options
########################
# Configure image backend.
#glance_backend_file: "yes"
glance_backend_ceph: "yes"
#glance_backend_vmware: "no"
#glance_backend_swift: "no"

##################
# Barbican options
##################
# Valid options are [ simple_crypto, p11_crypto ]
#barbican_crypto_plugin: "simple_crypto"
#barbican_library_path: "/usr/lib/libCryptoki2_64.so"

################
## Panko options
################
# Valid options are [ mongodb, mysql ]
#panko_database_type: "mysql"

#################
# Gnocchi options
#################
# Valid options are [ file, ceph ]
#gnocchi_backend_storage: "{{ 'ceph' if enable_ceph|bool else 'file' }}"

################################
# Cinder - Block Storage Options
################################
# Enable / disable Cinder backends
cinder_backend_ceph: "{{ enable_ceph }}"
#cinder_backend_vmwarevc_vmdk: "no"
cinder_volume_group: "cinder-volumes"

# Valid options are [ nfs, swift, ceph ]
cinder_backup_driver: "ceph"
#cinder_backup_share: ""
#cinder_backup_mount_options_nfs: ""

###################
# Designate options
###################
# Valid options are [ bind9 ]
#designate_backend: "bind9"
#designate_ns_record: "sample.openstack.org"

########################
# Nova - Compute Options
########################
nova_backend_ceph: "{{ enable_ceph }}"

# Valid options are [ qemu, kvm, vmware ]
nova_compute_virt_type: "kvm"

#################
# Hyper-V options
#################
# Hyper-V can be used as hypervisor
#hyperv_username: "user"
#hyperv_password: "password"
#vswitch_name: "vswitch"
# URL from which Nova Hyper-V MSI is downloaded
#nova_msi_url: "https://www.cloudbase.it/downloads/HyperVNovaCompute_Beta.msi"

#############################
# Horizon - Dashboard Options
#############################
#horizon_backend_database: "{{ enable_murano | bool }}"

######################################
# Manila - Shared File Systems Options
######################################
# HNAS backend configuration
#hnas_ip:
#hnas_user:
#hnas_password:
#hnas_evs_id:
#hnas_evs_ip:
#hnas_file_system_name:

################################
# Swift - Object Storage Options
################################
# Swift expects block devices to be available for storage. Two types of storage
# are supported: 1 - storage device with a special partition name and filesystem
# label, 2 - unpartitioned disk  with a filesystem. The label of this filesystem
# is used to detect the disk which Swift will be using.

# Swift support two matching modes, valid options are [ prefix, strict ]
#swift_devices_match_mode: "strict"

# This parameter defines matching pattern: if "strict" mode was selected,
# for swift_devices_match_mode then swift_device_name should specify the name of
# the special swift partition for example: "KOLLA_SWIFT_DATA", if "prefix" mode was
# selected then swift_devices_name should specify a pattern which would match to
# filesystems' labels prepared for swift.
#swift_devices_name: "KOLLA_SWIFT_DATA"

################################################
# Tempest - The OpenStack Integration Test Suite
################################################
# following value must be set when enable tempest
#tempest_image_id:
#tempest_flavor_ref_id:
#tempest_public_network_id:
#tempest_floating_network_name:

# tempest_image_alt_id: "{{ tempest_image_id }}"
# tempest_flavor_ref_alt_id: "{{ tempest_flavor_ref_id }}"

###################################
# VMware - OpenStack VMware support
###################################
#vmware_vcenter_host_ip:
#vmware_vcenter_host_username:
#vmware_vcenter_host_password:
#vmware_datastore_name:
#vmware_vcenter_name:
#vmware_vcenter_cluster_name:

Revision history for this message

José Donoso (jose.manuel.akainix) wrote on 2017-10-12:

I review the hypervisor stats and i see that i dont have free disk in my hypervisors

+----------------------+--------+
| Property | Value |
+----------------------+--------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 0 |
| free_disk_gb | 0 |
| free_ram_mb | 785220 |
| local_gb | 0 |
| local_gb_used | 0 |
| memory_mb | 786244 |
| memory_mb_used | 1024 |
| running_vms | 0 |
| vcpus | 96 |
| vcpus_used | 0 |
+----------------------+--------+

How can i change this if i am using ceph?

José Donoso (jose.manuel.akainix) on 2017-10-13

description:

updated

Revision history for this message

Javier Castillo (javcasalc) wrote on 2017-10-23:

In a lab deployment, I've got the same problem. I think it's related with ceph/cinder/glance. As result, my only lab hypervisor has "free_disk_gb = 0" (which is an error and should get ceph volume available disk) and it cannot deploy any instance:

$ openstack hypervisor stats show
+----------------------+-------+
| Field | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 0 |
| free_disk_gb | 0 |
| free_ram_mb | 9487 |
| local_gb | 0 |
| local_gb_used | 0 |
| memory_mb | 9999 |
| memory_mb_used | 512 |
| running_vms | 0 |
| vcpus | 6 |
| vcpus_used | 0 |
+----------------------+-------+

Revision history for this message

Javier Castillo (javcasalc) wrote on 2017-10-23:

Download full text (3.7 KiB)

Checking cinder_volume logs in one storage node, I get:

2017-10-23 08:49:07.768 735 INFO cinder.volume.manager [req-af1815ba-30f8-4a72-8097-756219536c65 - - - - -] Starting volume driver RBDDriver (1.2.0)
2017-10-23 08:49:07.770 735 DEBUG cinder.volume.drivers.rbd [req-af1815ba-30f8-4a72-8097-756219536c65 - - - - -] connecting to ceph (timeout=5). _do_conn /var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308
2017-10-23 08:49:07.794 735 DEBUG cinder.volume.drivers.rbd [req-af1815ba-30f8-4a72-8097-756219536c65 - - - - -] connecting to ceph (timeout=5). _do_conn /var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:308
2017-10-23 08:49:07.816 735 ERROR oslo_service.service [req-af1815ba-30f8-4a72-8097-756219536c65 - - - - -] Error starting thread.: IndexError: list index out of range
2017-10-23 08:49:07.816 735 ERROR oslo_service.service Traceback (most recent call last):
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
2017-10-23 08:49:07.816 735 ERROR oslo_service.service service.start()
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/service.py", line 242, in start
2017-10-23 08:49:07.816 735 ERROR oslo_service.service service_id=Service.service_id)
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 440, in init_host
2017-10-23 08:49:07.816 735 ERROR oslo_service.service self.driver.init_capabilities()
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/driver.py", line 704, in init_capabilities
2017-10-23 08:49:07.816 735 ERROR oslo_service.service stats = self.get_volume_stats(True)
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats
2017-10-23 08:49:07.816 735 ERROR oslo_service.service self._update_volume_stats()
2017-10-23 08:49:07.816 735 ERROR oslo_service.service File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, i...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.