Implement stable MORef for VMware

[kiran pawar]

Description
============
The "mo-ref" in VMware is not guaranteed to be stable. It can change after a recovery or simply un-registering and re-registering a vm. So making it stable is often necessary for the vcenter to be able to manage vms which were "lost" somehow.
Possible scenario is, when VMs reside on a ESXi host, which was down, and the ESXi host comes back, but has lost the vm. For the Stable VM refs, we need to restart the agent to clear out the cache. More problematic is the volume-ref as there is no fallback to the instance-uuid and we need to modify the db to reflect the new mo-ref.

Affected versions
=================
vsphere (all), openstack (all).

Steps to reproduce:
===================
For the VM:
1. Create an Instance
2. Stop the instance
3. Unregister the VM in vsphere
4. Register the VM in vsphere
5. Start the instance in openstack
6. An exception is raised

For the volume:
1. Create an instance with vmdk-volume attached
2. Unregister shadow-vm in vcenter
3. Register shadow-vm in vcenter
4. Detach the volume
5. An exception is raised

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/832164

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/832156
Committed: https://opendev.org/openstack/nova/commit/03fd208c562214476019ff2d7cc38f95c06e1348
Submitter: "Zuul (22348)"
Branch: master

commit 03fd208c562214476019ff2d7cc38f95c06e1348
Author: Fabian Wiesel <email address hidden>
Date: Sat Mar 5 09:29:11 2022 +0000

    VMware: Split out VMwareAPISession

    The VMwareAPISession object is not only used by the driver, but in
    practically all modules of vmwareapi. It reduces a bit the scope of
    the driver module itself.

    Partial-Bug: #1962771

    Change-Id: I4094b6031872bd3b5c871b9a82c7e01280a3352d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/832164
Committed: https://opendev.org/openstack/nova/commit/56055ede03fff633020e0f34fa4da5c8457a89c6
Submitter: "Zuul (22348)"
Branch: master

commit 56055ede03fff633020e0f34fa4da5c8457a89c6
Author: Fabian Wiesel <email address hidden>
Date: Sat Mar 5 13:53:18 2022 +0000

    VMware: StableMoRefProxy for moref recovery

    The vmwareapi driver uses Managed-Object references throughout the code
    with the assumption that they are stable. It is however a database id,
    which may change during the runtime of the compute node. e.g. If an
    instance is unregistered and re-registerd in the vcenter, the moref will
    change. By wrapping a moref in a proxy object, with an additional method
    to resolve the openstack object to a moref, we can hide those changes
    from a caller.

    MoRef implementation with closure - should ease the transition to stable
    mo-refs One simply has to pass the search function as a closure to the
    MoRef instance, and the very same method will be called when an
    exception is raised for the stored reference.

    Stable Volume refs - The connection_info['data'] contains the
    managed-object reference (moref) as well as the uuid of the volume.
    When the moref become invalid for some reason, we can recover it by
    searching for the volume-uuid as the `config.instanceUuid` attribute
    of the shadow-vm.

    Stable VM Ref - By encapsulating all the parameters for searching for
    the vm-ref again, we can move the retry logic to the session object,
    where we can try to recover the vm-ref should it result in a
    ManagedObjectNotFound exception.

    Use refs as index for fakedb - It was previously using the object-id
    to lookup an object, meaning that you couldn't pass a newly created
    Managed-object-reference like you could over the vmware-api. Now the
    lookup happens over the ref-id string, and in turn some functions
    were refactored to take that into account.

    Partial-Bug: #1962771

    Change-Id: I2a3ddf95b7fe07630855b06e732f8764efb13e91