Description
===========
In OpenStack release Xena with VMware vSphere I experience an issue while creating Nova instance with config drive.
Steps to reproduce
==================
1. build Kolla images with Neutron plugins according https://docs.openstack.org/kolla/xena/admin/image-building.html#neutron-plugins
$ kolla-build -b ubuntu --tag xena --config-file vmware-nsx.ini --push
2. deploy OpenStack controller using Kolla-ansible all-in-one inventory with globals.yml options customized (see below)
3. create Cirros image (qemu-img converted to VMDK), create flavor, create provider network type VLAN
4. create Nova instance
openstack server create \
--image cirros \
--flavor m1.tiny \
--network 'network1' \
--use-config-drive \
vm-$(date +%H%M%S)
Expected result
===============
Nova instance / VMware virtual machine with CD/DVD drive connected to file configdrive.iso in VM datastore path.
Actual result
=============
Nova instance in error state, VM deleted from VMware vSphere.
Logs & Configs
==============
I can provide more details if needed.
==> nova-compute.log <==
2023-05-02 19:54:41.461 8 DEBUG oslo_concurrency.processutils [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] Running cmd (subprocess): genisoimage -o /tmp/tmpxffhad19/configdrive.iso -ldots -allow-lowercase
-allow-multidot -l -publisher OpenStack Nova 24.2.1 -quiet -J -r -V config-2 /tmp/tmpvwpb0rsh execute
/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/processutils.py:384
2023-05-02 19:54:41.517 8 DEBUG oslo_concurrency.processutils [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] CMD "genisoimage -o /tmp/tmpxffhad19/configdrive.iso -ldots -allow-lowercase -allow-multidot -l -publisher
OpenStack Nova 24.2.1 -quiet -J -r -V config-2 /tmp/tmpvwpb0rsh" returned: 0 in 0.056s execute
/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_concurrency/processutils.py:422
2023-05-02 19:54:41.523 8 DEBUG nova.virt.vmwareapi.images [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] Uploading iso /tmp/tmpxffhad19/configdrive.iso to datastore
upload_iso_to_datastore /var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/images.py:216
2023-05-02 19:54:41.526 8 DEBUG oslo_vmware.rw_handles [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] Creating HTTP connection to write to file with size = 493568 and URL =
https://100.78.168.240:443/folder/10c093ea-5cb8-4cf8-8a70-156aed87621c/configdrive.iso?dcPath=%2FAT1&dsName=DS_LOC_VMC01_T2-01. _create_write_connection
/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_vmware/rw_handles.py:120
2023-05-02 19:54:41.565 8 DEBUG nova.virt.vmwareapi.images [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] Uploading iso of size : 493568 upload_iso_to_datastore
/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/images.py:228
2023-05-02 19:54:41.569 8 ERROR nova.virt.vmwareapi.vmops [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991
f07e4a8c218e4183a7b8e8d8e5153a13 - default default] [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] Creating config drive failed with error: 'utf-8' codec
can't decode byte 0xf1 in position 32848: invalid continuation byte: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 32848: invalid
continuation byte
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991 f07e4a8c218e4183a7b8e8d8e5153a13
- default default] [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] Instance failed to spawn: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in
position 32848: invalid continuation byte
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] Traceback (most recent call last):
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 2652, in _build_resources
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] yield resources
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 2412, in _build_and_run_instance
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] self.driver.spawn(context, instance, image_meta,
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/driver.py", line 532, in spawn
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] self._vmops.spawn(context, instance, image_meta, injected_files,
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/vmops.py", line 821, in spawn
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] self._configure_config_drive(
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/vmops.py", line 217, in _configure_config_drive
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] uploaded_iso_path = self._create_config_drive(context,
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/vmops.py", line 881, in _create_config_drive
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] LOG.error('Creating config drive failed with error: %s',
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] self.force_reraise()
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] raise self.value
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/vmops.py", line 870, in _create_config_drive
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] images.upload_iso_to_datastore(
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/vmwareapi/images.py", line 231, in upload_iso_to_datastore
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] data = iso_file.read(block_size)
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] File "/usr/lib/python3.8/codecs.py", line 322, in decode
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] (result, consumed) = self._buffer_decode(data, self.errors, final)
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 32848: invalid continuation byte
2023-05-02 19:54:41.576 8 ERROR nova.compute.manager [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c]
2023-05-02 19:54:41.602 8 INFO nova.compute.manager [req-c537ecba-e141-452a-b550-145fc33ad95e 7e9f152bbd104af6923cfdfadf957991 f07e4a8c218e4183a7b8e8d8e5153a13 - default default] [instance: 10c093ea-5cb8-4cf8-8a70-156aed87621c] Terminating instance
Environment
===========
Kolla:
pip install git+https://opendev.org/openstack/kolla@stable/xena
Kolla-ansible:
pip install git+https://opendev.org/openstack/kolla-ansible@stable/xena
OpenStack release Xena:
nova --version
17.6.0
openstack --version
openstack 5.6.2
VMware vSphere 6.7
--- begin of /etc/kolla/globals.yml ---
kolla_base_distro: "ubuntu"
nova_compute_virt_type: "vmware"
vmware_dvs_host_ip: <IP>
vmware_dvs_host_port: "443"
vmware_dvs_host_username: "<userName>"
vmware_dvs_dvs_name: "<dvsName>"
vmware_dvs_dhcp_override_mac: ""
enable_haproxy: "no"
enable_cinder: "yes"
glance_backend_file: "yes"
glance_backend_vmware: "no"
cinder_backend_vmwarevc_vmdk: "yes"
nova_compute_virt_type: "vmware"
vmware_vcenter_host_ip: <IP>
vmware_vcenter_host_username: <username>
vmware_vcenter_host_password: <password>
vmware_datastore_name: <dsName>
vmware_vcenter_name: <hostname>
vmware_vcenter_cluster_name: <clusterName>
--- end of /etc/kolla/globals.yml ---
The vmware driver is marked as experimental [1] because there is currently no CI (though some folks did step up to resurrect a vmware CI) and the code is in an unknown, possibly/probably buggy state. I don't necessarily want to mark this bug as Invalid just yet, because there is still some hope, but I want to set expectations that this bug may end up being closed and left unaddressed.
[1] https:/