instance_group_member entries not deleted when the instance deleted

Bug #1442098 reported by Attila Fazekas
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Charlotte Han
Liberty
Fix Released
Undecided
Vijay
Mitaka
Fix Released
Undecided
Matt Riedemann

Bug Description

Just the not deleted members needs to be selected, an instance group can gather many-many deleted instances during on his lifetime.

The selecting query contains a condition for omitting the deleted records:

SELECT instance_groups.created_at AS instance_groups_created_at, instance_groups.updated_at AS instance_groups_updated_at, instance_groups.deleted_at AS instance_groups_deleted_at, instance_groups.deleted AS instance_groups_deleted, instance_groups.id AS instance_groups_id, instance_groups.user_id AS instance_groups_user_id, instance_groups.project_id AS instance_groups_project_id, instance_groups.uuid AS instance_groups_uuid, instance_groups.name AS instance_groups_name, instance_group_policy_1.created_at AS instance_group_policy_1_created_at, instance_group_policy_1.updated_at AS instance_group_policy_1_updated_at, instance_group_policy_1.deleted_at AS instance_group_policy_1_deleted_at, instance_group_policy_1.deleted AS instance_group_policy_1_deleted, instance_group_policy_1.id AS instance_group_policy_1_id, instance_group_policy_1.policy AS instance_group_policy_1_policy, instance_group_policy_1.group_id AS instance_group_policy_1_group_id, instance_group_member_1.created_at AS instance_group_member_1_created_at, instance_group_member_1.updated_at AS instance_group_member_1_updated_at, instance_group_member_1.deleted_at AS instance_group_member_1_deleted_at, instance_group_member_1.deleted AS instance_group_member_1_deleted, instance_group_member_1.id AS instance_group_member_1_id, instance_group_member_1.instance_id AS instance_group_member_1_instance_id, instance_group_member_1.group_id AS instance_group_member_1_group_id FROM instance_groups LEFT OUTER JOIN instance_group_policy AS instance_group_policy_1 ON instance_groups.id = instance_group_policy_1.group_id AND instance_group_policy_1.deleted = 0 AND instance_groups.deleted = 0 LEFT OUTER JOIN instance_group_member AS instance_group_member_1 ON instance_groups.id = instance_group_member_1.group_id AND instance_group_member_1.deleted = 0 AND instance_groups.deleted = 0 WHERE instance_groups.deleted = 0 AND instance_groups.project_id = '6da55626d6a04f4c99980dc17d34235f';

(Captured at $nova server-group-list)

But actually nova fetches the deleted records because the `deleted` field is 0,
even if the instance already deleted.

For figuring out the instance is actually deleted the nova API issues other otherwise not needed queries.

The instance_group_member records actually set to deleted only when instance_group deleted.

show create table instance_group_member;

CREATE TABLE `instance_group_member` (
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  `deleted_at` datetime DEFAULT NULL,
  `deleted` int(11) DEFAULT NULL,
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `instance_id` varchar(255) DEFAULT NULL,
  `group_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `group_id` (`group_id`),
  KEY `instance_group_member_instance_idx` (`instance_id`),
  CONSTRAINT `instance_group_member_ibfk_1` FOREIGN KEY (`group_id`) REFERENCES `instance_groups` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8

1, Please delete the instance_group_member records when the instance gets deleted.
2, Please add (`deleted`,`group_id`) BTREE index as combined index, in this way it will be usable in other situations as well, for example when only a single group's members is needed.

Alex Xu (xuhj)
Changed in nova:
assignee: nobody → Alex Xu (xuhj)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/173664

Changed in nova:
status: New → In Progress
tags: added: db
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Alex Xu (<email address hidden>) on branch: master
Review: https://review.openstack.org/173664
Reason: I didn't have time working on this, abandon it, leave other people free to pick it.

Alex Xu (xuhj)
Changed in nova:
assignee: Alex Xu (xuhj) → nobody
Changed in nova:
assignee: nobody → Syed Ahsan Shamim Zaidi (ahsanmohsin04)
Changed in nova:
assignee: Syed Ahsan Shamim Zaidi (ahsanmohsin04) → nobody
Changed in nova:
status: In Progress → Confirmed
Charlotte Han (hanrong)
Changed in nova:
assignee: nobody → Rong Han ZTE (hanrong)
Revision history for this message
Charlotte Han (hanrong) wrote :
Download full text (9.3 KiB)

The bug is reproduced.

It's correct for using RESTFUL API.
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+
| Id | Name | Policies | Members | Metadata |
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+
| 211a62d0-de78-4311-8250-c014fed46f0e | antiaffinitygroup | [u'anti-affinity'] | [u'5c37820b-d9e5-49b6-ad3c-9845f1667224', u'd1da3837-a5c5-45a4-8b2b-bc53559fcc8e'] | {} |
+--------------------------------------+-------------------+--------------------+------------------------------------------------------------------------------------+----------+
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------+
| 7c32f761-3f49-46b5-86c6-1cbfe8a4e0d5 | test | ERROR | - | NOSTATE | |
| 1964dae7-89bb-45d4-af90-ddba48c135a7 | test2 | ERROR | - | NOSTATE | |
| 5c37820b-d9e5-49b6-ad3c-9845f1667224 | test2 | ERROR | - | NOSTATE | |
| d1da3837-a5c5-45a4-8b2b-bc53559fcc8e | test2 | ERROR | - | NOSTATE | |
+--------------------------------------+-------+--------+------------+-------------+----------+
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$ nova delete 7c32f761-3f49-46b5-86c6-1cbfe8a4e0d5
Request to delete server 7c32f761-3f49-46b5-86c6-1cbfe8a4e0d5 has been accepted.
[tecs@tecs nova(keystone_admin)]$ nova delete 1964dae7-89bb-45d4-af90-ddba48c135a7
Request to delete server 1964dae7-89bb-45d4-af90-ddba48c135a7 has been accepted.
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$ nova list
+--------------------------------------+-------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------+
| 5c37820b-d9e5-49b6-ad3c-9845f1667224 | test2 | ERROR | - | NOSTATE | |
| d1da3837-a5c5-45a4-8b2b-bc53559fcc8e | test2 | ERROR | - | NOSTATE | |
+--------------------------------------+-------+--------+------------+-------------+----------+
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$
[tecs@tecs nova(keystone_admin)]$ nova server-group-list
+--------------------------------------+-------------------+--------------------+-...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/289392

Changed in nova:
status: Confirmed → In Progress
Matt Riedemann (mriedem)
tags: added: server-groups
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/289392
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e2f4370b04598833939f2b869e7bac11c02a4921
Submitter: Jenkins
Branch: master

commit e2f4370b04598833939f2b869e7bac11c02a4921
Author: zte-hanrong <email address hidden>
Date: Mon Mar 7 23:21:32 2016 +0800

    Soft delete instance group member when delete instance

    Currently after instance deleted, the instance is still as member
    of instance group. This patch make sure the instance will removed
    from instance group when execute instance_destroy db call.

    Closes-Bug: #1442098

    Change-Id: I8cae3e5c317f0797944ecf3bea21c571ff24d9cf

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308490

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/308490
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ef1c5dbcf3451c081356b45633220ad3969d5b70
Submitter: Jenkins
Branch: stable/mitaka

commit ef1c5dbcf3451c081356b45633220ad3969d5b70
Author: zte-hanrong <email address hidden>
Date: Mon Mar 7 23:21:32 2016 +0800

    Soft delete instance group member when delete instance

    Currently after instance deleted, the instance is still as member
    of instance group. This patch make sure the instance will removed
    from instance group when execute instance_destroy db call.

    Closes-Bug: #1442098

    Change-Id: I8cae3e5c317f0797944ecf3bea21c571ff24d9cf
    (cherry picked from commit e2f4370b04598833939f2b869e7bac11c02a4921)

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 13.1.0

This issue was fixed in the openstack/nova 13.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.