Nova unable to find valid hosts caused failure creating servers

[Erich Cordoba]

Brief Description
-----------------
Nova is unable to create a server, the openstack server show command shows a 'No valid host was found' error.

Severity
--------
Major

Steps to Reproduce
------------------
Try to create a server with:

openstack server create --image cirros --flavor f1.small --nic net-id=9551e11d-a7f0-4a17-b36b-226a1f453f33 vm-cirros-1

Expected Behavior
------------------
The server should be created, up and running.

Actual Behavior
----------------
The server remains in ERROR state. See the output of openstack server show vm-cirros-1 --format yaml

OS-DCF:diskConfig: MANUAL
OS-EXT-AZ:availability_zone: ''
OS-EXT-SRV-ATTR:host: null
OS-EXT-SRV-ATTR:hypervisor_hostname: null
OS-EXT-SRV-ATTR:instance_name: instance-0000000a
OS-EXT-STS:power_state: NOSTATE
OS-EXT-STS:task_state: null
OS-EXT-STS:vm_state: error
OS-SRV-USG:launched_at: null
OS-SRV-USG:terminated_at: null
accessIPv4: ''
accessIPv6: ''
addresses: ''
config_drive: ''
created: '2019-06-10T17:37:06Z'
fault:
  code: 500
  created: '2019-06-10T17:37:07Z'
  details: " File \"/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py\"\
    , line 1326, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n\
    \ File \"/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py\"\
    , line 780, in _schedule_instances\n return_alternates=return_alternates)\n\
    \ File \"/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py\"\
    , line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n\
    \ File \"/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\"\
    , line 160, in select_destinations\n return cctxt.call(ctxt, 'select_destinations',\
    \ **msg_args)\n File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\"\
    , line 178, in call\n retry=self.retry)\n File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py\"\
    , line 128, in _send\n retry=retry)\n File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\"\
    , line 645, in send\n call_monitor_timeout, retry=retry)\n File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\"\
    , line 636, in _send\n raise result\n"
  message: No valid host was found. There are not enough hosts available.
flavor: f1.small (73b04076-5484-45df-ab5f-cfb8e9765005)
hostId: ''
id: adf2799c-d633-4d04-a19d-d5cb79588691
image: cirros (0a1e049c-f359-4b6c-8260-e3320cafae78)
key_name: null
name: vm-cirros-1
project_id: dc3b3d8cb4444ab8bafa2046a2fcc3e6
properties: ''
status: ERROR
updated: '2019-06-10T17:37:06Z'
user_id: 63a6cc7cfa6b4617a0656d03132dd157
volumes_attached: ''

Reproducibility
---------------
100%

System Configuration
--------------------
So far in Simplex and External storage baremetal.

Branch/Pull Time/Commit
-----------------------
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190609T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="136"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-09 23:30:00 +0000"

Timestamp/Logs
--------------
See attached logs.

Test Activity
-------------
Sanity

[Erich Cordoba (ericho)]

It turns out that running `openstack compute service list` I got:

+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| 2 | nova-consoleauth | nova-consoleauth-6fdfb5c47f-tnbv8 | internal | enabled | up | 2019-06-10T18:07:04.000000 |
| 3 | nova-scheduler | nova-scheduler-6cb694cbb6-xsjcj | internal | enabled | up | 2019-06-10T18:07:09.000000 |
| 4 | nova-conductor | nova-conductor-54cc9bfd99-j87ww | internal | enabled | down | 2019-06-10T17:22:45.000000 |
| 5 | nova-compute | controller-0 | nova | disabled | up | 2019-06-10T18:07:05.000000 |
| 7 | nova-conductor | nova-conductor-54cc9bfd99-wj8rp | internal | enabled | up | 2019-06-10T18:07:06.000000 |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+

Notice the 'disabled' state for nova-compute. After doing a openstack compute service set --enable controller-0 nova-compute I was able to create the server.

controller-0:~$ openstack server list
+--------------------------------------+-------------+--------+------------------------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------+--------+------------------------+--------+----------+
| 14394307-b8b2-4161-b1e7-b258f3d6efe3 | vm-cirros-1 | ACTIVE | network-1=192.168.0.48 | cirros | f1.small |
+--------------------------------------+-------------+--------+------------------------+--------+----------+

[Erich Cordoba (ericho)]

This is the output of openstack compute service list in a standard external storage system:

controller-0:~$ openstack compute service list --format yaml

controller-0:~$ openstack compute service list
+----+------------------+-----------------------------------+----------+----------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+-----------------------------------+----------+----------+-------+----------------------------+
| 8 | nova-consoleauth | nova-consoleauth-59fcb9fc9c-j8lzd | internal | enabled | up | 2019-06-10T17:19:20.000000 |
| 11 | nova-scheduler | nova-scheduler-6986b44f56-l8s69 | internal | enabled | down | 2019-06-10T14:41:08.000000 |
| 14 | nova-scheduler | nova-scheduler-6986b44f56-f26xh | internal | enabled | up | 2019-06-10T17:19:18.000000 |
| 17 | nova-consoleauth | nova-consoleauth-59fcb9fc9c-c42pr | internal | enabled | down | 2019-06-10T14:41:10.000000 |
| 20 | nova-conductor | nova-conductor-6cffff977-fgmff | internal | enabled | up | 2019-06-10T17:19:17.000000 |
| 23 | nova-conductor | nova-conductor-6cffff977-wzf9t | internal | enabled | down | 2019-06-10T14:41:09.000000 |
| 26 | nova-compute | compute-0 | nova | disabled | down | 2019-06-10T14:41:29.000000 |
| 29 | nova-compute | compute-1 | nova | disabled | up | 2019-06-10T17:19:16.000000 |
+----+------------------+-----------------------------------+----------+----------+-------+----------------------------+

[Ghada Khalil (gkhalil)]

Please provide the nfv-vim and nova-compute logs leading up to the failure. A full collect log from the system would be ideal.

[Yang Liu (yliu12)]

This is likely to be the same issue as 1832254 which is no longer seen since master 20190611T000451Z.

[Ghada Khalil (gkhalil)]

Closing as Invalid since the issue is not reproducible and marking as a duplicate of https://bugs.launchpad.net/starlingx/+bug/1832254

It's unclear how this issue was introduced and how it was resolved.