Migrate VM failed when anti-affinity group was removed.

[bryan]

Description
===========
Migrate VMs failed when anti-affinity group was removed from VMs.
When anti-affinity group was removed, group_hosts of request_spec is set to an empty list (here's what I got when debugging nova-conductor: {'limits': {}, 'group_policies': set([u'anti-affinity']), 'ignore_hosts': [u'test-deploy7'], 'group_hosts': set([]), 'group_updated': True, 'scheduler_hints': {u'group': u'groupId'}}), then in nova.objects.request_spec.RequestSpec#_populate_group_info line 188: hosts = list(filter_properties.get('group_hosts')), and "TypeError: 'NoneType' object is not iterable" exception was raised.

Steps to reproduce
==================
1. Create an anti-affinity group named A;
2. Create two VMs and assign these two VM into the anti-affinity group A created in previous step;
3. Remove anti-affinity group A;
4. Migrate these two VMs, and all of them got failed.

Expected result
===============
These two VMs were migrated successfully.

Actual result
=============
Migration of these two VMs got failed.

Environment
===========
1. Exact version of OpenStack you are running. See the following
    OpenStack Newton.

2. Which hypervisor did you use?
   Libvirt

2. Which storage type did you use?
    Ceph

3. Which networking type did you use?
   Neutron with OpenVSwitch

Logs & Configs
==============
2017-03-23 15:15:44.081 41242 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: c8f3a7be34c145a0a6921770994d5fae __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions [req-16b1e2ec-9a77-48dc-ade0-980650973089 142d8663efce464c89811c63e45bd82e f21a9c86d7114bf99c711f4874d80474 - 9f95b9967b894c928880feb32fad1d0d 9f95b9967b894c928880feb32fad1d0d] Unexpected exception in API method
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 52, in _migrate
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions self.compute_api.resize(req.environ['nova.context'], instance)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 173, in _wrapped
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return fn(self, context, instance, *args, **kwargs)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return f(self, context, instance, *args, **kw)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3033, in resize
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions request_spec=request_spec)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 192, in resize_instance
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions request_spec=request_spec)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 317, in migrate_server
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return cctxt.call(context, 'migrate_server', **kw)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions retry=self.retry)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions retry=retry)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions raise result
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions TypeError: 'NoneType' object is not iterable
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions res = self.dispatcher.dispatch(message)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return self._do_dispatch(endpoint, method, ctxt, args)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions result = func(ctxt, **new_args)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return function(self, context, *args, **kwargs)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 220, in migrate_server
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions reservations, clean_shutdown, request_spec)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 283, in _cold_migrate
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions updates, ex, legacy_spec)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions self.force_reraise()
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 252, in _cold_migrate
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions task.execute()
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions self.rollback()
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions self.force_reraise()
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return original(self)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions return self._execute()
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 54, in _execute
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions self.instance.availability_zone)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 385, in from_components
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions spec_obj._populate_group_info(filter_properties)
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/request_spec.py", line 188, in _populate_group_info
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions members = list(filter_properties.get('group_members'))
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions TypeError: 'NoneType' object is not iterable
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.082 41242 ERROR nova.api.openstack.extensions
2017-03-23 15:15:44.084 41242 INFO nova.api.openstack.wsgi [req-16b1e2ec-9a77-48dc-ade0-980650973089 142d8663efce464c89811c63e45bd82e f21a9c86d7114bf99c711f4874d80474 - 9f95b9967b894c928880feb32fad1d0d 9f95b9967b894c928880feb32fad1d0d] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/451664

[bryan (lcsong)]

Here's the patch fixing this bug: https://review.openstack.org/#/c/451664/

[Matt Riedemann (mriedem)]

Looking at the code, the RequestSpec.from_components method is called without an instance_group value (None is passed in):

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L50

Which means the RequestSpec instance group is generated from the legacy filter properties:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L384

And the group_members value comes out of the filter properties dict here:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L188

And since the group was delete, there are no group members, so we get the TypeError.

Before calling RequestSpec.from_components, conductor calls this method to set instance group information in the request spec:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L43

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/scheduler/utils.py#L335

However, we'd only set group_members in filter_properties if we were able to lookup the group from the database:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/scheduler/utils.py#L353

And if you deleted the group, I'm not sure how we'd get there.

In other words, I'm not sure how this is True:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L183

Unless the filter_properties dict has some old information in it?

Ah yes, it's due to this:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/objects/request_spec.py#L300

Called starting here:

https://github.com/openstack/nova/blob/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/conductor/tasks/migrate.py#L42

That was fixed here for bug 1719730:

https://github.com/openstack/nova/commit/9d6632a67d91fb3c5145c14ac38011e919d6d8c0

So it looks like that needs to be backported, although newton is nearly end of life at this point.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/451664