VM in ERROR state due to not valid host found

[Maria Guadalupe Perez Ibara]

Brief Description
-----------------
VM in ERROR state due to not valid host found

Severity
--------
Major

Steps to Reproduce
------------------
openstack flavor create --ram 2048 --disk 20 --vcpus 1 --public --id auto f1.small
openstack subnet create --network network1 --subnet-range 192.168.1.0/24 subnet1
openstack volume create --size 20 --image cirros --bootable vol-cirros-1
openstack image create --file /home/sysadmin//cirros-0.4.0-x86_64-disk.qcow2 --disk-format qcow2 --public cirros
openstack server create --image cirros --flavor f1.small --nic net-id=87a2e82c-dbdc-4828-bda1-f3e2b5e09a65 vm-cirros-1

Expected Behavior
------------------
VM should be created without problem.

Actual Behavior
----------------
the output of openstack server list shows:
+--------------------------------------+-------------+--------+----------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-------------+--------+----------+--------+----------+
| ef41a6d4-e837-4a5b-9284-63b7cf24e332 | vm-cirros-1 | ERROR | | cirros | f1.small |
+--------------------------------------+-------------+--------+----------+--------+----------+

Reproducibility
---------------
100 %

System Configuration
--------------------
simplex, duplex, 2+2, 2+2+2 virtual

Branch/Pull Time/Commit
-----------------------
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190620T013000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="153"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-20 01:30:00 +0000"

Last Pass
---------
6/13/2019

Timestamp/Logs
--------------

| fault | {u'message': u'No valid host was found. ', u'code': 500, u'details': u' File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 1326, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/conductor/manager.py", line 780, in _schedule_instances\n return_alternates=return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/var/lib/openstack/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call\n retry=self.retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send\n retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send\n call_monitor_timeout, retry=retry)\n File "/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send\n raise result\n', u'created': u'2019-06-20T22:57:17Z'} |
| flavor | f1.small (56b94963-81fd-4f1b-9e27-2750cfe9cd14)

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

I believe this is the same underlying issue as reported in:
https://bugs.launchpad.net/starlingx/+bug/1833497

Assigning to Zhipeng to review and confirm

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 release gating as this impacts sanity on virtual environment.

[Gerry Kopec (gerry-kopec)]

Looks like nova-scheduler couldn't find any available hosts from placement:
{"log":"2019-06-20 22:57:15.792 1 INFO nova.scheduler.manager [req-498a23fb-a08c-4a57-b415-96f355d98089 8a94107e714b4c90b20bf5186af7cd65 c4139aa3b0e846ff92af3a912d7dc6dd - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.\n","stream":"stdout","time":"2019-06-20T22:57:15.793587566Z"}

That is due to the fact that there are no placement pods running (and nova-placement is no longer started per placement removal). So agree this is the same issue as: https://bugs.launchpad.net/starlingx/+bug/1833497

[Ghada Khalil (gkhalil)]

Thanks for investigating Gerry.
Marking as a duplicate of https://bugs.launchpad.net/starlingx/+bug/1833497

Fix merged on 2019-06-20 and should be in the June 21 cengn build.
https://review.opendev.org/666491