nova-manage cell_v2 verify_instance returns a valid instance mapping even after the instance is deleted/archived

[Surya Seetharaman]

Although nova-manage cell_v2 verify_instance is used to check if the provided instance is correctly mapped to a cell or not, this should not be returning a valid mapping message if the instance itself is deleted. It should return an error message saying 'The instance does not exist'.

Steps to reproduce :

1. Create an instance :

-> nova boot --image 831bb8a0-9305-4cd7-b985-cbdadfb5d3db --flavor m1.nano test
-> nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| aec6eb34-6aaf-4883-8285-348d40fdac87 | test | ACTIVE | - | Running | public=2001:db8::4, 172.24.4.9 |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+

2. Delete the instance :

-> nova delete test
Request to delete server test has been accepted.
-> nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+
+--------------------------------------+--------+--------+------------+-------------+---------------------------------+

3. Verify Instance :

-> nova-manage cell_v2 verify_instance --uuid aec6eb34-6aaf-4883-8285-348d40fdac87
Instance aec6eb34-6aaf-4883-8285-348d40fdac87 is in cell: cell5 (c5ccba5d-1a45-4739-a5dd-d665a1b19301)

Basically the message that we get is misleading for a deleted instance. This is because verify_instance queries the instance_mappings table which maintains a mapping of the deleted instances as well.

[Surya Seetharaman (tssurya)]

There are other related bugs like below dealing with the existence of information about deleted instances in the instance_mapping table :

1. https://bugs.launchpad.net/nova/+bug/1722451
2. https://bugs.launchpad.net/nova/+bug/1679941

However, even if we decide to do a purge operation (https://bugs.launchpad.net/nova/+bug/1722451/comments/1) , the output of verify_instance for a deleted instance should not be dependent on the purge, IMHO.

[melanie witt (melwitt)]

End users of the REST API (with appropriate policy permission) can 'nova list' deleted instances. So it's not straightforward to decide whether or not to show that the instance is mapped. It *is* mapped and it can be seen by the end user if they can list deleted instances.

[Surya Seetharaman (tssurya)]

Following the discussion on the IRC channel, we have thus decided to do a follow up deletion of those instance_mappings from the nova-api database which is not done currently, once the corresponding rows are archived from the instances table using nova-manage db archive_deleted_rows.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/515034

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/515034
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=32fd58813f8247641a6b574b5f01528b29d48b76
Submitter: Zuul
Branch: master

commit 32fd58813f8247641a6b574b5f01528b29d48b76
Author: Surya Seetharaman <email address hidden>
Date: Wed Oct 25 13:43:43 2017 +0200

    cleanup mapping/reqspec after archive instance

    This patch aims at deleting the records of the archived instances from
    the instance_mappings and request_specs tables in the API database
    immediately following their archival from instances to shadow_instances
    table. So upon running the 'nova-manage db archive_deleted_rows' command
    the records of the archived instances will be automatically removed from
    the instance_mappings and request_specs tables as well. A warning has
    also been added to fix the issue of 'nova-manage verify_instance'
    returning a valid instance mapping even after the instance is deleted.

    The patch also adds InstanceMappingList.destory_bulk() and
    RequestSpec.destroy_bulk() methods for ease of bulk deletion of records.

    Change-Id: I483701a55576c245d091ff086b32081b392f746e
    Closes-Bug: #1724621
    Closes-Bug: #1678056

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/580775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/580775
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=70de423255fd01822188bb9082a0c0cc1c8ec2d0
Submitter: Zuul
Branch: stable/pike

commit 70de423255fd01822188bb9082a0c0cc1c8ec2d0
Author: Surya Seetharaman <email address hidden>
Date: Wed Oct 25 13:43:43 2017 +0200

    cleanup mapping/reqspec after archive instance

    This patch aims at deleting the records of the archived instances from
    the instance_mappings and request_specs tables in the API database
    immediately following their archival from instances to shadow_instances
    table. So upon running the 'nova-manage db archive_deleted_rows' command
    the records of the archived instances will be automatically removed from
    the instance_mappings and request_specs tables as well. A warning has
    also been added to fix the issue of 'nova-manage verify_instance'
    returning a valid instance mapping even after the instance is deleted.

    The patch also adds InstanceMappingList.destory_bulk() and
    RequestSpec.destroy_bulk() methods for ease of bulk deletion of records.

    Change-Id: I483701a55576c245d091ff086b32081b392f746e
    Closes-Bug: #1724621
    Closes-Bug: #1678056
    (cherry picked from commit 32fd58813f8247641a6b574b5f01528b29d48b76)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.