Instance stuck spawning (win_2012 dedicated image)

[Wendy Mitchell]

STX: Instance stuck spawning (win_2012 dedicated image)

Brief Description
-----------------
Instantiation of vm (from win_2012 dedicated image)

Severity
--------
Major

Steps to Reproduce
------------------
Create image
glance --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://<address>:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne image-create --property os_type=windows --name win_2012 --container-format bare --wait 1 --disk-format qcow2 --file /home/wrsroot//images/win2012r2.qcow2 --visibility public

Create Flavor eg. 3ae5df3b-90dc-4eaf-bb46-f33294c4d603

Extra specs
hw:cpu_policy": "dedicated", "aggregate_instance_extra_specs:storage": "remote"
1024 RAM
disk 13

[2018-10-10 04:55:01,231]
Boot instance
nova --os-username 'tenant2' --os-password 'Li69nux*' --os-project-name tenant2 --os-auth-url http://<address>:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne boot --image 1bedf9ca-526c-4ccf-8bff-623ab6cce653 --user-data /home/wrsroot//userdatawin_2012_2nic_cloud_init_if_conf.sh --key-name keypair-tenant2 --flavor 3ae5df3b-90dc-4eaf-bb46-f33294c4d603 --nic net-id=d13f0e54-21d9-4dae-99cf-31fd944ca5aa,vif-model=virtio --nic net-id=0a96c56d-0030-405c-868f-c557d3d70ec4,vif-model=virtio tenant2-win_2012-dedicated-migrate-130 --poll'

Expected Behavior
------------------
Expected instantiation if schedule/claim successful.

Actual Behavior
----------------
Claim successful on compute-1 but stuck in spawning

nova-compute.log

2018-10-10 04:55:07.027 62491 INFO nova.virt.libvirt.driver [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] [instance: d2eceed2-6d4f-4928-8327-ac7e4a292fd4] Creating image
..

Created image .part completed @ 2018-10-10 04:55:52.021

2018-10-10 04:55:07.048 62491 INFO nova.virt.libvirt.imagebackend [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] Creating image 13eac24b909b2737dce857dc3b07e4d85793f458 ...
2018-10-10 04:55:07.250 62491 INFO nova.virt.images [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] virt.images.fetch(): acquiring disk_op_sema
2018-10-10 04:55:07.251 62491 INFO nova.virt.images [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] fetch: Downloading 1bedf9ca-526c-4ccf-8bff-623ab6cce653 to /etc/nova/instances/_base/13eac24b909b2737dce857dc3b07e4d85793f458.part
2018-10-10 04:55:07.251 62491 INFO nova.image.api [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] download: Downloading 1bedf9ca-526c-4ccf-8bff-623ab6cce653 to /etc/nova/instances/_base/13eac24b909b2737dce857dc3b07e4d85793f458.part
2018-10-10 04:55:52.021 62491 INFO nova.virt.images [req-b8e3ddb6-e68e-4b79-8bbf-8a0cb3ce1fd4 202f820d5c9a49dcb635abdf60fa871a 64064f8da7604c85a5c9b9f5a0654038 - default default] fetch: Downloading /etc/nova/instances/_base/13eac24b909b2737dce857dc3b07e4d85793f458.part completed

Instance starts building here but remains in task_state=spawning

 2018-10-10 04:55:55.951 62491 INFO nova.compute.resource_tracker [req-4fd220b2-d5c9-427f-ba6a-ffb6a1d5d4ef - - - - -] [instance: d2eceed2-6d4f-4928-8327-ac7e4a292fd4] sign=1, id=instance-000000a7, name=tenant2-win_2012-dedicated-migrate-130, vm_mode=building,

Reproducibility
---------------
yes

System Configuration
--------------------
controller/storage/compute system

Branch/Pull Time/Commit
-----------------------
master as of 2018-10-08_01-52-01

Timestamp/Logs
--------------

[Ghada Khalil (gkhalil)]

Targeting stx.2019.03 as this appears to be an intermittent issue as confirmed by the designer investigating the issue

[Frank Miller (sensfan22)]

Requesting assistance to debug this issue. While it is intermittent, it appears to occur if enough attempts are made when launching a VM on a dedicated storage config. Initial investigation seem to indicate the VM launch is getting stuck when doing the qemu-img convert.
eg:
qemu-img convert -O raw /etc/nova/instances/_base/5de8269d7cc450ef88c74688ec5d6b598e85fb44 rbd:ephemeral/0f4e065f-e8c9-4aac-9a2b-6c1c82a7cac9_disk:conf=/etc/ceph/ceph.conf

[Bruce Jones (brucej)]

Please re-test now that we have rebased to openstack master.

[Austin Sun (sunausti)]

as similar with https://bugs.launchpad.net/starlingx/+bug/1804017, re-test in latest code base , can not reproduce this issue.

[Austin Sun (sunausti)]

if you can still reproduce in latest version, please re-open this issue.

[Wendy Mitchell (wmitchellwr)]

This no longer appears to be an issue with containerized load.

Successfully configured storage lab with 3 remote worker nodes
Ran the following win2012 testcase without issue.

nova/test_migrate_vms.py::test_migrate_vm_various_guest[win_2012-3-1024-dedicated-image]

Was also able to manually create win2012 glance images and spawned instance without issue.

yow-cgcs-wildcat-92-98
BUILD_ID="20190410T013000Z"