server group members are not deleted on failed server create overquota

[Matt Riedemann]

When creating instances in the database in the compute API, if we fail after creating them we attempt to delete the instances from the DB here:

https://github.com/openstack/nova/blob/af7e83fef3bc2c005c581587e9230a4070f8feb9/nova/compute/api.py#L1033

However, if there is a failure it's ignored and we continue and just re-raise the exception.

The instances can fail to delete because of a referential constraint on the block device mappings created here:

https://github.com/openstack/nova/blob/af7e83fef3bc2c005c581587e9230a4070f8feb9/nova/compute/api.py#L1471

So if we don't delete those first, we can't cleanup the instances. You can recreate this by changing CONF.quota_server_group_members=0 and trying to boot a server into a server group.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/304929

[Matt Riedemann (mriedem)]

This can prevent archiving/purging instances so bumping to medium severity and marking it for backport.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304929
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5674e7646d106751b27d191e3334d9e6ebe9ab1b
Submitter: Jenkins
Branch: master

commit 5674e7646d106751b27d191e3334d9e6ebe9ab1b
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 12 22:09:16 2016 -0400

    Properly clean up BDMs when _provision_instances fails

    _provision_instances calls create_db_entry_for_new_instance
    which creates the instance and block device mappings in the
    database.

    The instance is added to the instances list which is used
    in the global exception block at the bottom of _provision_instances
    to destroy any instances created. A failure that can trigger
    this cleanup attempt after the instance and BDMs are created
    is when checking server group member count fails with OverQuota.

    The problem is that we fail to (soft) delete the block device mappings
    that we created. Since there is a foreign key constraint between
    the block_device_mapping and instances tables in the database,
    when we try to archive (copy soft deleted things to shadow tables
    and then hard-delete them) the deleted instance it will fail on
    a referential constraint error due to the BDM(s) which weren't deleted.

    We can fix this by deleting the BDMs when deleting the instance just
    like we do for other reference tables.

    A functional test is added to demonstrate the failure and fix which
    also has the nice benefit of functionally testing the server group
    member overquota error handling.

    Change-Id: Ida66a93031046bafcf30c95ca232fb6382c2597b
    Closes-Bug: #1569641

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308491

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308491
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6fce7d3bc9df7157edfc762d2dbd4800d834dd51
Submitter: Jenkins
Branch: stable/mitaka

commit 6fce7d3bc9df7157edfc762d2dbd4800d834dd51
Author: Matt Riedemann <email address hidden>
Date: Tue Apr 12 22:09:16 2016 -0400

    Properly clean up BDMs when _provision_instances fails

    _provision_instances calls create_db_entry_for_new_instance
    which creates the instance and block device mappings in the
    database.

    The instance is added to the instances list which is used
    in the global exception block at the bottom of _provision_instances
    to destroy any instances created. A failure that can trigger
    this cleanup attempt after the instance and BDMs are created
    is when checking server group member count fails with OverQuota.

    The problem is that we fail to (soft) delete the block device mappings
    that we created. Since there is a foreign key constraint between
    the block_device_mapping and instances tables in the database,
    when we try to archive (copy soft deleted things to shadow tables
    and then hard-delete them) the deleted instance it will fail on
    a referential constraint error due to the BDM(s) which weren't deleted.

    We can fix this by deleting the BDMs when deleting the instance just
    like we do for other reference tables.

    A functional test is added to demonstrate the failure and fix which
    also has the nice benefit of functionally testing the server group
    member overquota error handling.

    Change-Id: Ida66a93031046bafcf30c95ca232fb6382c2597b
    Closes-Bug: #1569641
    (cherry picked from commit 5674e7646d106751b27d191e3334d9e6ebe9ab1b)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 13.1.1 release.