_destroy_evacuated_instances fails randomly with high number of instances

[Luis Fernández Álvarez]

In our production environment (2013.2.1), we're facing a random error thrown while starting nova-compute in Hyper-V nodes.

The following exception is thrown while calling '_destroy_evacuated_instances':

16:30:58.802 7248 ERROR nova.openstack.common.threadgroup [-] 'NoneType' object is not iterable
2014-03-05 16:30:58.802 7248 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
(...)
2014-03-05 16:30:58.802 7248 TRACE nova.openstack.common.threadgroup File "C:\Python27\lib\site-packages\nova\compute\manager.py", line 532, in _get_instances_on_driver
2014-03-05 16:30:58.802 7248 TRACE nova.openstack.common.threadgroup name_map = dict((instance['name'], instance) for instance in instances)
2014-03-05 16:30:58.802 7248 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object is not iterable

Full trace: http://paste.openstack.org/show/73243/

Our first guess is that this problem is related with number of instances in our deployment (~3000), they're all fetched in order to check evacuated instances (as Hyper-V is not implementing "list_instance_uuids").

In the case of KVM, this error is not happening as it's using a smarter method to get this list based on the UUID of the instances.

Although this is being reported using Hyper-V, it's a problem that could occur in other drivers not implementing "list_instance_uuids"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/79961

[John Garbutt (johngarbutt)]

We use milestones only at the very end of the milestone to track bugs that should block the release of the milestone. Please use tags not milestones to track bugs, try things like: my_bug_topic_milestone_name

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79961
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=73da55e4ef626283ae58a97c7ad89854ec77daa3
Submitter: Jenkins
Branch: master

commit 73da55e4ef626283ae58a97c7ad89854ec77daa3
Author: Alessandro Pilotti <email address hidden>
Date: Wed Mar 12 16:53:32 2014 +0200

    Adds list_instance_uuids to the Hyper-V driver

    In case of large number of servers, the _destroy_evacuated_instances
    fails randomly. Implementing list_instance_uuids solves the issue.

    Co-Authored-By: Robert Tingirica <email address hidden>
    Co-Authored-By: Ionut Balutoiu <email address hidden>
    Change-Id: I9c58168c012b342bb5dfa0c62a7c39a327f442b3
    Closes-bug: #1291364

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/120390

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120390
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9015410ec95fe7dc23fb3cb2be432915a9733cbd
Submitter: Jenkins
Branch: stable/icehouse

commit 9015410ec95fe7dc23fb3cb2be432915a9733cbd
Author: Alessandro Pilotti <email address hidden>
Date: Wed Mar 12 16:53:32 2014 +0200

    Adds list_instance_uuids to the Hyper-V driver

    In case of large number of servers, the _destroy_evacuated_instances
    fails randomly. Implementing list_instance_uuids solves the issue.

    Co-Authored-By: Robert Tingirica <email address hidden>
    Co-Authored-By: Ionut Balutoiu <email address hidden>
    Closes-bug: #1291364
    (cherry picked from commit 73da55e4ef626283ae58a97c7ad89854ec77daa3)

    Conflicts:
     nova/tests/virt/hyperv/test_vmops.py
     nova/tests/virt/hyperv/test_vmutils.py
     nova/tests/virt/hyperv/test_vmutilsv2.py
     nova/virt/hyperv/vmutils.py

    Change-Id: I9c58168c012b342bb5dfa0c62a7c39a327f442b3