nova stuck at boot Server building... 0% complete after 900 seconds

[Chris Winnicki]

Brief Description
-----------------
nova stuck at boot Server building... 0% complete after 900 seconds

Severity
--------
Major: Unable to boot VMs

Steps to Reproduce
------------------
Install 2+3 system (2 controllers and 3 computes)
Boot a VM, ex: (refer to Actual Behavior)

Expected Behavior
------------------
VMs should successfully boot withing 900 seconds

Actual Behavior
----------------
VM fails to boot, stuck in scheduling/booting

Details: CLI 'nova --os-username 'tenant2' --os-password 'Li69nux*' --os-project-name tenant2 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --key-name keypair-tenant2 --boot-volume b267b615-7b74-4baf-94eb-1827ff333d45 --flavor 846fb0db-e52f-4ada-89a5-0eece81f2340 --nic net-id=44371032-ff2c-4b37-b842-2efc0673120b --nic net-id=1a731806-6882-4460-b8c8-d80102aca5da --nic net-id=dc610974-f487-4bd3-880e-66718d3daf5d tenant2-multiports_pci_base-12 --poll' failed to execute. Output: +--------------------------------------+--------------------------------------------------------------------------------------------------------+
| Property | Valu |
+--------------------------------------+--------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zon | |
| OS-EXT-STS:power_stat | 0 |
| OS-EXT-STS:task_stat | scheduling |
| OS-EXT-STS:vm_stat | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | tKfAuAp9q4D |
| config_driv | |
| created | 2019-05-12T00:28:53Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:cpu_policy": "dedicated", "hw:mem_page_size": "2048", "hw:pci_numa_affinity_policy": "preferred"} |
| flavor:original_nam | dedicated |
| flavor:ram | 2048 |
| flavor:swap | 0 |
| flavor:vcpus | 2 |
| hostId | |
| id | ac4d653a-61b4-46a7-911a-5bf5e5f5ab42 |
| imag | Attempt to boot from volume - no image supplied |
| key_nam | keypair-tenant2 |
| locked | Fals |
| metadata | {} |
| nam | tenant2-multiports_pci_base-12 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | BUILD |
| tags | [] |
| tenant_id | 709962ae9eb84a99a68326bcfaf905f |
| trusted_image_certificates | - |
| updated | 2019-05-12T00:28:53Z |
| user_id | 4e0444e9e38041acadee74d4be6dfae0 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------+

Server building... 0% complete
Server building... 0% complete
Server building... 0% complete
Server building... 0% complete

Reproducibility
---------------
100% reproducible on build: 20190508T013000Z

System Configuration
--------------------
Node Config: 2+3
Lab: R720_3_7

Branch/Pull Time/Commit
-----------------------
Load: 20190508T013000Z
Job: STX_build_master_master
Build Server: starlingx_mirror
Software Version: 19.01

Last Pass
---------
20190503T013000Z

Timestamp/Logs
--------------
2019-05-12T00:28:53Z

Test Activity
-------------
Networking Regression / PCI Robustness

[Ghada Khalil (gkhalil)]

This is a nova upstream issue. A commit was recently back-ported to the nova stein branch which is causing this issue.
The issue has been reported to the nova team via:
https://bugs.launchpad.net/nova/+bug/1829062

This launchpad will be used to switch the stx docker image builds from nova stein to the stx nova fork. This was planned anyway for the current release of stx to pick up other key content which did not make it to stein.

[Ghada Khalil (gkhalil)]

Just to further clarify, the stx nova fork is based on an older stein snapshot, so it doesn't have this issue.

The plan is for the stx nova fork to be rebased at a regular cadence. The stx.disto.openstack team will need to monitor the nova bug above to ensure a fix is available before the next rebase of the fork.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/659335

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/659335
Committed: https://git.openstack.org/cgit/starlingx/upstream/commit/?id=1702e2308475ed67abe10a0d80cc5a11e4ac06b0
Submitter: Zuul
Branch: master

commit 1702e2308475ed67abe10a0d80cc5a11e4ac06b0
Author: Don Penney <email address hidden>
Date: Wed May 15 12:19:00 2019 -0400

    Build stx-nova from staging branch

    Update the stx-nova.stable_docker_image directives file
    to pull nova source from the stx-nova staging branch,
    using stx/stein.1 feature branch.

    Change-Id: I3529e263ed77d1847d655f72de8979726fc62c09
    Closes-Bug: 1829060
    Signed-off-by: Don Penney <email address hidden>