Nova VMwareapi Resize of Volume Backed server fails

Bug #2055411 reported by Fabian Wiesel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Unassigned

Bug Description

Description
===========
More specifically the following tempest test in master fails:
tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server

Steps to reproduce
==================
* Install Devstack from master
* Run tempest test `tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server`

Expected result
===============
The test succeeds.

Actual result
=============
What happened instead of the expected result?
How did the issue look like?

Environment
===========
1. Git 1858cf18b940b3636e54eb5aafaf4050bdd02939 (master). So essentially this:
 https://review.opendev.org/c/openstack/nova/+/909474
As instance creation is impossible without that patch.

2. Which hypervisor did you use? What's the version of that?

vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

2. Which storage type did you use?

vmdk on NFS 4.1

3. Which networking type did you use?

networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

Logs & Configs
==============

Can be found here: http://openstack-ci-logs.global.cloud.sap/openstack/nova/1858cf18b940b3636e54eb5aafaf4050bdd02939/index.html

The critical exception for this bug report is (abbreviated and reformatted for clarity):
----
 req-7aa5ded6-ea97-4010-93c8-9e39389cbfe0 tempest-ServerActionsTestOtherA-839537081
[ 865.017199] env[58735]: ERROR nova.compute.manager [instance: b4d9131c-fc91-4fd4-813b-13b4bdfe1647]
Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 10856, in _error_out_instance_on_exception
    yield
  File "/opt/stack/nova/nova/compute/manager.py", line 6096, in _resize_instance
    disk_info = self.driver.migrate_disk_and_power_off(
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 263, in migrate_disk_and_power_off
    return self._vmops.migrate_disk_and_power_off(context, instance,
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1467, in migrate_disk_and_power_off
    self._resize_disk(instance, vm_ref, vmdk, flavor)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1398, in _resize_disk
    self._volumeops.detach_disk_from_vm(vm_ref, instance, vmdk.device)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 121, in detach_disk_from_vm
    disk_key = device.key
AttributeError: 'NoneType' object has no attribute 'key'

---

The bug is actually in the function `nova.virt.vmwareapi.vm_util.get_vmdk_info` here:
https://opendev.org/openstack/nova/src/branch/master/nova/virt/vmwareapi/vm_util.py#L690

The code works with the assumption, that the root-disk is named as the instance.
This assumption breaks in several cases, but most for this test-case, the root volume is actually a cinder volume.
It will also break when the the disk gets migrated to another datastore, either through a live-migration with no shared storage, or simply automatically with SDRS..

I have an alternative implementation here: https://github.com/sapcc/nova/blob/stable/xena-m3/nova/virt/vmwareapi/vm_util.py#L997-L1034
I'll provide a bug fix from it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/910627

Changed in nova:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.