Nova No valid host was found

[Yvonne Ding]

Brief Description
-----------------
exception.NoValidHost: "No valid host was found" is thrown when boot a vm.

Severity
--------
Major

Steps to Reproduce
------------------
1. Create a flavor with 2 vcpus
2. Add following extra specs: {'hw:cpu_policy': 'dedicated'}
3. Boot a opensuse_12 vm with above flavor from image

TC-name:
TestVariousGuests::test_nova_actions_various_guest[opensuse_12-dedicated-image-pause-unpause-suspend-resume-stop-start-auto_recover]
TestVariousGuests::test_nova_actions_various_guest[rhel_6-dedicated-image-pause-unpause-suspend-resume-stop-start-auto_recover]
TestVariousGuests::test_nova_actions_various_guest[win_2012-dedicated-volume-pause-unpause-suspend-resume-stop-start-auto_recover]
TestVariousGuests::test_nova_actions_various_guest[win_2016-dedicated-image-pause-unpause-suspend-resume-stop-start-auto_recover]
test_lock_with_vms.py::TestLockWithVMs::test_lock_with_vms

Expected Behavior
-----------------
Boot a Vm succeed

Actual Behavior
----------------
Boot a VM failed

Reproducibility
---------------
reproducible

System Configuration
--------------------
Regular standard 2+2

Lab-name:
ip_1_4

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20210226T024233Z"

Timestamp/Logs
--------------
[2021-03-07 09:07:11,680] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show d9d04f07-1d5b-4fc9-9c3a-039cef18854e'
[2021-03-07 09:07:13,673] 436 DEBUG MainThread ssh.expect :: Output:
| Field | Value

| fault | {u'message': u'No valid host was found. ', u'code': 500, u'details': u'Traceback (most recent call last):\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 1463, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 870, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call\n transport_options=self.transport_options)\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send\n transport_options=transport_options)\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/var/lib/openstack/lib/python3.6/site-packages/nova/scheduler/manager.py", line 200, in select_destinations\n raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n', u'created': u'2021-03-07T09:07:06Z'} |
| flavor | dedicated (79ff9706-9103-4f9e-9a5c-399292a82c85) |
| hostId | |
| id | d9d04f07-1d5b-4fc9-9c3a-039cef18854e |
| image | opensuse_12 (d91b3011-4de1-4c17-8a44-3850367571ce) |
| key_name | keypair-tenant1 |
| name | tenant1-nova_actions-opensuse_12-image-72 |
| project_id | 6aa476fc5cbd40f9bdb55a01e9a1a4cb |
| properties | |
| status | ERROR |
| updated | 2021-03-07T09:07:06Z |
| user_id | 2f649ef7745e4be7b41280558d48904a |
| volumes_attached | |

> raise exceptions.VMOperationFailed(output)
E utils.exceptions.VMOperationFailed: VM operation failed.
E Details: +--------------------------------------+-------------------------------------------------------------+
E | Property | Value |
E +--------------------------------------+-------------------------------------------------------------+
E | OS-DCF:diskConfig | MANUAL |
E | OS-EXT-AZ:availability_zone | |
E | OS-EXT-STS:power_state | 0 |
E | OS-EXT-STS:task_state | scheduling |
E | OS-EXT-STS:vm_state | building |
E | OS-SRV-USG:launched_at | - |
E | OS-SRV-USG:terminated_at | - |
E | accessIPv4 | |
E | accessIPv6 | |
E | adminPass | kU48ZzybpGdC |
E | config_drive | |
E | created | 2021-03-07T09:07:06Z |
E | description | - |
E | flavor:disk | 21 |
E | flavor:ephemeral | 0 |
E | flavor:extra_specs | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "large"} |
E | flavor:original_name | dedicated |
E | flavor:ram | 1024 |
E | flavor:swap | 0 |
E | flavor:vcpus | 2 |
E | hostId | |
E | id | d9d04f07-1d5b-4fc9-9c3a-039cef18854e |
E | image | opensuse_12 (d91b3011-4de1-4c17-8a44-3850367571ce) |
E | key_name | keypair-tenant1 |
E | locked | False |
E | locked_reason | - |
E | metadata | {} |
E | name | tenant1-nova_actions-opensuse_12-image-72 |
E | os-extended-volumes:volumes_attached | [] |
E | progress | 0 |
E | security_groups | default |
E | server_groups | [] |
E | status | BUILD |
E | tags | [] |
E | tenant_id | 6aa476fc5cbd40f9bdb55a01e9a1a4cb |
E | trusted_image_certificates | - |
E | updated | 2021-03-07T09:07:06Z |
E | user_id | 2f649ef7745e4be7b41280558d48904a |
E +--------------------------------------+-------------------------------------------------------------+
E
E
Server building... 0% complete
E Error building server
E ERROR (ResourceInErrorState):

logs of .tar and TIS_AUTOMATION.log as below,
https://files.starlingx.kube.cengn.ca/launchpad/
Note: above url is currently not accessible

Test Activity
-------------
STX Nova regression

[Yvonne Ding (yding)]

Please note above .tar and .log will be moved to cengn log portal when it is back to up. Currently cengn log portal is unavailable.

[Frank Miller (sensfan22)]

Screening:
This may be an issue with the config used in the lab. Marking stx.6.0 gating but not critical to address for stx.5.0.

[Austin Sun (sunausti)]

Hi Douglas and Frank:
    do we have conclusion for this issue ? is it nat box issue?

[Douglas Lopes Pereira (douglaspereira)]

Hi Austin, we were not able to reproduce it in our lab. Can we close it and reopen it if this is observed in the future?

[Austin Sun (sunausti)]

ok. Thanks.