instance_system_metadata rows not being deleted

[Phil Day]

Deleting an instance no longer seems to delete the associated instance_system_metadata rows in the DB.

To reproduce in Devstack:
- Create an instance
- Delete it
- Look at the associated DB rows:

mysql> select * from instances where uuid="9ad5d7c7-306f-44ff-84a2-06b9af7d5d36" \G ;
*************************** 1. row ***************************
              created_at: 2013-09-16 10:18:05
              updated_at: 2013-09-16 10:21:44
              deleted_at: 2013-09-16 10:21:44
                      id: 7
             internal_id: NULL
                 user_id: eb50e969766f4cfcb392d307d1178a9e
              project_id: 852d7be63c7c4540856b38be6226ff49
               image_ref: 28b52a16-4a62-497c-8c83-837cdcf6bf66
               kernel_id: 86f818dc-d836-4b1f-aae5-386bc594a746
              ramdisk_id: 41e93f5e-06c5-49eb-9ed4-3b5e10f56d0a
            launch_index: 0
                key_name: NULL
                key_data: NULL
             power_state: 1
                vm_state: deleted
               memory_mb: 512
                   vcpus: 1
                hostname: phil
                    host: vm-reap
               user_data: NULL
          reservation_id: r-8arbuy8m
            scheduled_at: 2013-09-16 10:18:06
             launched_at: 2013-09-16 10:18:10
           terminated_at: 2013-09-16 10:21:44
            display_name: phil
     display_description: phil
       availability_zone: NULL
                  locked: 0
                 os_type: NULL
             launched_on: vm-reap
        instance_type_id: 2
                 vm_mode: NULL
                    uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
            architecture: NULL
        root_device_name: /dev/vda
            access_ip_v4: NULL
            access_ip_v6: NULL
            config_drive:
              task_state: NULL
default_ephemeral_device: NULL
     default_swap_device: NULL
                progress: 0
        auto_disk_config: 0
      shutdown_terminate: 0
       disable_terminate: 0
                 root_gb: 1
            ephemeral_gb: 0
               cell_name: NULL
                    node: vm-reap.novalocal
                 deleted: 7
               locked_by: NULL
                 cleaned: 0
1 row in set (0.00 sec)

ERROR:
No query specified

mysql> select * from instance_metadata where instance_uuid="9ad5d7c7-306f-44ff-84a2-06b9af7d5d36" \G ;
Empty set (0.00 sec)

ERROR:
No query specified

mysql> select * from instance_system_metadata where instance_uuid="9ad5d7c7-306f-44ff-84a2-06b9af7d5d36" \G ;
*************************** 1. row ***************************
   created_at: 2013-09-16 10:18:05
   updated_at: NULL
   deleted_at: NULL
           id: 83
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: image_kernel_id
        value: 86f818dc-d836-4b1f-aae5-386bc594a746
      deleted: 0
*************************** 2. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 84
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_memory_mb
        value: 512
      deleted: 0
*************************** 3. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 85
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_swap
        value: 0
      deleted: 0
*************************** 4. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 86
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_vcpu_weight
        value: NULL
      deleted: 0
*************************** 5. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 87
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_root_gb
        value: 1
      deleted: 0
*************************** 6. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 88
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_name
        value: m1.tiny
      deleted: 0
*************************** 7. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 89
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: image_ramdisk_id
        value: 41e93f5e-06c5-49eb-9ed4-3b5e10f56d0a
      deleted: 0
*************************** 8. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 90
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_id
        value: 2
      deleted: 0
*************************** 9. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 91
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_ephemeral_gb
        value: 0
      deleted: 0
*************************** 10. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 92
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_rxtx_factor
        value: 1
      deleted: 0
*************************** 11. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 93
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_flavorid
        value: 1
      deleted: 0
*************************** 12. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 94
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: instance_type_vcpus
        value: 1
      deleted: 0
*************************** 13. row ***************************
   created_at: 2013-09-16 10:18:06
   updated_at: NULL
   deleted_at: NULL
           id: 95
instance_uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
          key: image_base_image_ref
        value: 28b52a16-4a62-497c-8c83-837cdcf6bf66
      deleted: 0
13 rows in set (0.00 sec)

[Hans Lindgren (hanlind)]

This looks like it is intentional, see bug 1185190.

[Phil Day (philip-day)]

Thanks for the pointer. I don't accept however that having the instance_system_metadata table grow for eve on a large system is an acceptable workaround - so I'd like this bug to remain open. At the moment what's been left in the system is a poor workaround that threatens the stability of large systems - and so a better fix to the issue of reading deleted data needs to be found.

[David Ripton (dripton)]

This is one of the causes of bug 1183523.

We have a general problem with instance metadata never being deleted. I'd really like to see this fixed, but I don't think it will happen for Havana. Maybe Icehouse.

[Thang Pham (thang-pham)]

Would it make sense to put some parameter in nova.conf to set how long to keep the instance history, e.g. instance_system_metadata_keepalive = 365, where the value is the number of days? That way, you can delete it instantly if it is set to zero, or keep it as long as necessary.

[jichenjc (jichenjc)]

looks to me the #2 mentioned in bug 1183523 is gone because now
the audit will not read the deleted instances any more
#1 is still there, Thang's suggestion might be a good start

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/109201

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/110875

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/110875
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/109201
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Fix proposed again.

[Chris Friesen (cbf123)]

Is bug 1275479 a duplicate of this?

[jichenjc (jichenjc)]

yes, I think so

[Feng Xi Yan (yanfengxi)]

https://review.openstack.org/#/c/110875/
and
https://review.openstack.org/#/c/109201/
are merged in master branch.

I think they should also be back ported into kilo, juno and icehouse.

[jichenjc (jichenjc)]

Maybe only Kilo
since it's not a critical bug or security bug, thoughts?

[Feng Xi Yan (yanfengxi)]

Yes, makes sense.

Best Regard:)
Bruce Yan

Yan， Fengxi （闫凤喜）
Openstack Platform Team
IBM China Systems & Technology Lab, Beijing
E-Mail: <email address hidden>
Tel: [phone] Notes: Feng Xi FX Yan/China/IBM
Address: 3BW239, Ring Building. No.28 Building, ZhongGuanCun Software
Park,No.8
DongBeiWang West Road, ShangDi, Haidian District, Beijing, P.R.China

From: Chen CH Ji/China/IBM@IBMCN
To: Feng Xi BJ Yan/China/IBM@IBMCN
Date: 2015/07/09 14:50
Subject: [Bug 1226049] Re: instance_system_metadata rows not being
            deleted
Sent by: <email address hidden>

Maybe only Kilo
since it's not a critical bug or security bug, thoughts?

--
You received this bug notification because you are subscribed to a
duplicate bug report (1388756).
https://bugs.launchpad.net/bugs/1226049

Title:
  instance_system_metadata rows not being deleted

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Deleting an instance no longer seems to delete the associated
  instance_system_metadata rows in the DB.

  To reproduce in Devstack:
  - Create an instance
  - Delete it
  - Look at the associated DB rows:

  mysql> select * from instances where
uuid="9ad5d7c7-306f-44ff-84a2-06b9af7d5d36" \G ;
  *************************** 1. row ***************************
                created_at: 2013-09-16 10:18:05
                updated_at: 2013-09-16 10:21:44
                deleted_at: 2013-09-16 10:21:44
                        id: 7
               internal_id: NULL
                   user_id: eb50e969766f4cfcb392d307d1178a9e
                project_id: 852d7be63c7c4540856b38be6226ff49
                 image_ref: 28b52a16-4a62-497c-8c83-837cdcf6bf66
                 kernel_id: 86f818dc-d836-4b1f-aae5-386bc594a746
                ramdisk_id: 41e93f5e-06c5-49eb-9ed4-3b5e10f56d0a
              launch_index: 0
                  key_name: NULL
                  key_data: NULL
               power_state: 1
                  vm_state: deleted
                 memory_mb: 512
                     vcpus: 1
                  hostname: phil
                      host: vm-reap
                 user_data: NULL
            reservation_id: r-8arbuy8m
              scheduled_at: 2013-09-16 10:18:06
               launched_at: 2013-09-16 10:18:10
             terminated_at: 2013-09-16 10:21:44
              display_name: phil
       display_description: phil
         availability_zone: NULL
                    locked: 0
                   os_type: NULL
               launched_on: vm-reap
          instance_type_id: 2
                   vm_mode: NULL
                      uuid: 9ad5d7c7-306f-44ff-84a2-06b9af7d5d36
              architecture: NULL
          root_device_name: /dev/vda
              access_ip_v4: NULL
              access_ip_v6: NULL
              config_drive:
                task_state: NULL
  default_ephemeral_device: NULL
       default_swap_device: NULL
                  progress: 0
          auto_disk_config: 0
        shutdown_terminate: 0
         disable_terminate: 0
                   root_gb: 1
              ephemeral_gb: 0
                ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/204447

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/204448

[OpenStack Infra (hudson-openstack)]

Change abandoned by jichenjc (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/204447

[OpenStack Infra (hudson-openstack)]

Change abandoned by jichenjc (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/204448

[John Garbutt (johngarbutt)]

seems we have patches merged on master for this, so I have marked this as completed

The kilo work can be tracked separately, in the usual ways.