VMware: resize fails when there is more than one compute node

[Gary Kotton]

Doing a nova resize on an instance when using the vsphere driver will cause the instance to go in to error state.

The problem is that the scheduler will pick another host to spin up a new resized instance and when the user confirms nova will fail because its looking for the instance on the old compute.

Here is the traceback.

2014-07-16 18:14:55.271 13228 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
2014-07-16 18:14:55.271 13228 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
2014-07-16 18:14:55.273 13228 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420
2014-07-16 18:14:55.299 13228 ERROR nova.openstack.common.rpc.amqp [req-3cae0e1d-2cf4-4da7-9aac-0ea5279b829d cherkasj 37af63b6867d4fe38ac312ca626ce186] Exception during message handling
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp **args)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp payload)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2683, in confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp do_confirm_resize(context, instance, migration_id)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2680, in do_confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp migration=migration)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2707, in _confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp network_info)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in confirm_migration
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp _vmops = self._get_vmops_for_compute_node(instance['node'])
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 567, in _get_vmops_for_compute_node
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp resource = self._get_resource_for_node(nodename)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 559, in _get_resource_for_node
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp raise exception.NotFound(msg)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp NotFound: The resource domain-c1122(compute02) does not exist
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp
2014-07-16 18:14:57.595 13228 DEBUG nova.openstack.common.vmware.api [-] Waiting for function _invoke_api to return. func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:120
2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking _invoke_api; retry count is 0. _func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:83
2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking method <module 'nova.virt.vmwareapi.vim_util' from '/usr/lib/python2.7/dist-packages/nova/virt/vmwareap

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/108225

[Gary Kotton (garyk)]

The problem is that certain operations update the instance node. This is validated to check that it matched the cluster configuration on the compute node. This is wrong as the compute node just needs to perform a VC opertaion.

[John Garbutt (johngarbutt)]

This only affects VMware, so downgrading to High. Please see:
https://wiki.openstack.org/wiki/Bugs#Importance

[John Garbutt (johngarbutt)]

Doesn't seem like this should block J-2, so remove J-2 target for now, please shout on IRC if this is the wrong call.

[Kanagaraj Manickam (kanagaraj-manickam)]

Hi Gary,

I have investigated this issue and found an different solution here:

in nova.compute.api, on resize() method, by using resize_allow_on_same_host = True, following code could help to solve the issue:

       if not CONF.allow_resize_to_same_host:
            filter_properties['ignore_hosts'].append(instance['host'])
        else:
            if instance['host'] != instance['node']:
                filter_properties['force_nodes'] = [instance['node']]

[Tracy Jones (tjones-i)]

as this only affects 1 hypervisor by definition it cannot be critical

[Michael Still (mikal)]

Is anyone working on this bug? If not we might need to untarget it from rc1.

[Gary Kotton (garyk)]

A patch has been in review - https://review.openstack.org/108225.

[Gary Kotton (garyk)]

Hopefully we can have this for RC2.

[Joe Gordon (jogo)]

please stop marking this is critical.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/108225
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8e4a9156f4dccf003970848c28b8a9d15c55212d
Submitter: Jenkins
Branch: master

commit 8e4a9156f4dccf003970848c28b8a9d15c55212d
Author: Gary Kotton <email address hidden>
Date: Sat Jul 19 23:24:54 2014 -0700

    VMware: fix exception when multiple compute nodes are running

    A number of operations in the VMwareVCDriver class first validate
    that instances node is the cluster that is mapped to the compute
    node. This is problematic when the compute nodes have different
    configurations, for example, each compute node is mapped to a
    different cluster.

    In this case many operations that are just performing instance operations
    will fail. This patch ensure that all instance operations that do not
    require a cluster or volume will make use of the base _vmops class.
    This is due to the fact that it only requires the instance details
    to interface with the VC and there are no specific cluster operations.

    Change-Id: I2bc38a480f2feb12ea41e7d28f80b29dd49a79b8
    Closes-bug: #1345460

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/137386

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137386
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=acefbcb20408ec6ee1fe666524a2a20449969f1f
Submitter: Jenkins
Branch: stable/juno

commit acefbcb20408ec6ee1fe666524a2a20449969f1f
Author: Gary Kotton <email address hidden>
Date: Sat Jul 19 23:24:54 2014 -0700

    VMware: fix exception when multiple compute nodes are running

    A number of operations in the VMwareVCDriver class first validate
    that instances node is the cluster that is mapped to the compute
    node. This is problematic when the compute nodes have different
    configurations, for example, each compute node is mapped to a
    different cluster.

    In this case many operations that are just performing instance operations
    will fail. This patch ensure that all instance operations that do not
    require a cluster or volume will make use of the base _vmops class.
    This is due to the fact that it only requires the instance details
    to interface with the VC and there are no specific cluster operations.

    Change-Id: I2bc38a480f2feb12ea41e7d28f80b29dd49a79b8
    Closes-bug: #1345460
    (cherry picked from commit 8e4a9156f4dccf003970848c28b8a9d15c55212d)