nova's server group API returns deleted instances as members

[Amrith Kumar]

trove gate fails with error in wait_for_delete_non_affinity_master

This is a hard failure that has started occurring in the last couple of days.

AssertionError: Found left-over server group: <ServerGroup: 34e6dcaa-58f9-4aa8-a959-9d6dc8a6be6d>

Error first seen 2017-04-04.

[Amrith Kumar (amrith)]

The behavior that is causing this problem is illustrated from the command line (see illustration.txt). The issue appears to be that Nova is (now) returning deleted instances in a server group, something that it likely didn't do in the past.

[Matt Riedemann (mriedem)]

My guess would be something related to this recent change:

https://review.openstack.org/#/c/443293/

I haven't debugged this in detail though.

[Matt Riedemann (mriedem)]

I think the regression is probably because we don't actually delete instance_mappings records when we delete an instance, which is another known bug:

https://bugs.launchpad.net/nova/+bug/1679941

So when we delete an instance, it's gone from the main cell database query but the instance mapping isn't deleted, so we're still going to be returning those group members since we're using instance mappings now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/457097

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/457097
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e463d4f4383976927dad0092c2bff9b88ad22fe
Submitter: Jenkins
Branch: master

commit 3e463d4f4383976927dad0092c2bff9b88ad22fe
Author: Matt Riedemann <email address hidden>
Date: Sun Apr 16 21:55:27 2017 +0000

    Revert "Make server_groups determine deleted-ness from InstanceMappingList"

    This reverts commit 0af57a9d4be86afe75c986de667cbd9750017f64.

    As reported in bug 1679941, we don't delete instance_mappings records when
    we delete instances from the cells, so we can't really rely on the
    InstanceMappingList at this point to tell if an instance exists or not, since
    the instance might have been deleted but the mapping hasn't.

    Change-Id: Id75868ab9bef5136930d6bc33e197473b2c19977
    Closes-Bug: #1682845

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.0.0b2 development milestone.