OpenStack Compute (nova)

backup instance won't be able to delete all images exceeding rotation limit, if user mistakenly deletes one of the backup image

  1. Series newton
  2. Bug #1634773
Bug #1634773 reported by Bhagyashri Shewale
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Bhagyashri Shewale
Newton
Fix Committed
Low
Lee Yarwood

Bug Description

If user have already backed up instance to few numbers and then execute
backup api with rotation 1, then nova will delete the previously
created images exceeding rotation limit.

During deleting these images, if user mistakenly deletes one of the
image in advance, then backup instance won't be able to delete all
images exceeding rotation limit causing api failure.

Steps to reproduce:

 1. Create instance

  $ nova boot --flavor <flavor_id> --image <image_id> <server_name>

  $ nova boot --flavor 1 --image cirros-0.3.4-x86_64-uec test

  +--------------------------------------+------+--------+------------+-------------+---------------------------------+
  | ID | Name | Status | Task State | Power State | Networks |
  +--------------------------------------+------+--------+------------+-------------+---------------------------------+
  | a9e41bc5-516c-48f1-bed8-2475650835c8 | test | ACTIVE | - | Running | public=2001:db8::8, 172.24.4.12 |
  +--------------------------------------+------+--------+------------+-------------+---------------------------------+

 2. Create 5 instance backup using the backup api(hit below command 5 times)

  $ nova backup <server> <name> <backup-type> <rotation>

  $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 5

 3. Take list of backup images

  $ glance image-list

  +--------------------------------------+---------------------------------+
  | ID | Name |
  +--------------------------------------+---------------------------------+
  | c33790be-df8b-4740-8311-0fa787bfb087 | cirros-0.3.4-x86_64-uec |
  | de43a375-6d6d-4a30-856e-7a68db083834 | cirros-0.3.4-x86_64-uec-kernel |
  | 0a6c800e-6002-4f2f-a2b9-0542e0f2fac0 | cirros-0.3.4-x86_64-uec-ramdisk |
  | fc0d2030-1cec-4780-9199-e99c63da0005 | test_bk |
  | a9d7dc78-acab-4e4e-b03d-c7da9cb5ef00 | test_bk |
  | 937061b4-51c7-4835-b9b7-d63464665a27 | test_bk |
  | 41e25c26-c61e-421c-b170-a39539bcbc78 | test_bk |
  | a15ae46a-a3c3-49a8-be75-1aee724fa4e9 | test_bk |
  +--------------------------------------+---------------------------------+

 4. Add debug point using pdb.set_trace() in nova.compute.manager._rotate_backups method [1]

 5. Delete one of the backup images from above list

  $ glance image-delete fc0d2030-1cec-4780-9199-e99c63da0005

 6. Create the insatnce backup by passing the rotation parameter value as 1.

  $ nova backup a9e41bc5-516c-48f1-bed8-2475650835c8 test_bk daily 1

 7. Checked the n-cpu logs

2016-10-17 15:59:38.978 ERROR oslo_messaging.rpc.server [req-99fcf837-514a-4a80-a19e-c8ac2f26768f admin admin] Exception during message handling
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server function_name, call_dict, binary)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 179, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server instance=instance)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3001, in backup_instance
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self._rotate_backups(context, instance, backup_type, rotation)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3145, in _rotate_backups
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server self.image_api.delete(context, image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/api.py", line 141, in delete
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server return session.delete(context, image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 765, in delete
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server raise exception.ImageNotFound(image_id=image_id)
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server ImageNotFound: Image fc0d2030-1cec-4780-9199-e99c63da0005 could not be found.
2016-10-17 15:59:38.978 TRACE oslo_messaging.rpc.server

[1] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3139

Tags: compute
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/388519

Changed in nova:
assignee: nobody → Bhagyashri Shewale (bhagyashri-shewale)
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Bhagyashri Shewale (bhagyashri-shewale) → Michael Still (mikal)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Michael Still (mikal) → Bhagyashri Shewale (bhagyashri-shewale)
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Low
tags: added: compute
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Bhagyashri Shewale (bhagyashri-shewale) → melanie witt (melwitt)
melanie witt (melwitt)
Changed in nova:
assignee: melanie witt (melwitt) → Bhagyashri Shewale (bhagyashri-shewale)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/388519
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cc5d3437cabab4688df8492e071ea8c597311113
Submitter: Jenkins
Branch: master

commit cc5d3437cabab4688df8492e071ea8c597311113
Author: bhagyashris <email address hidden>
Date: Mon Oct 17 19:29:21 2016 +0530

    Handle ImageNotFound exception during instance backup

    If user have already backed up instance to few numbers and then
    execute backup api with rotation 1, then nova will delete the
    previously created images exceeding rotation limit.
    During deleting these images, if user mistakenly deletes one of the
    image in advance, then backup instance won't be able to delete all
    images exceeding rotation limit causing api failure.

    This patch handles ImageNotFound exception during deleting backup
    images, logs a warning message and continues deleting all of the
    remaining images.

    Closes-Bug: #1634773
    Change-Id: Ib999138ddbbcd62e0c99a69e5c6d37b4e49f871f

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409715

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b2

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/409715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0080a12361b480f5b87ebdb62285b4100550d0b1
Submitter: Jenkins
Branch: stable/newton

commit 0080a12361b480f5b87ebdb62285b4100550d0b1
Author: bhagyashris <email address hidden>
Date: Mon Oct 17 19:29:21 2016 +0530

    Handle ImageNotFound exception during instance backup

    If user have already backed up instance to few numbers and then
    execute backup api with rotation 1, then nova will delete the
    previously created images exceeding rotation limit.
    During deleting these images, if user mistakenly deletes one of the
    image in advance, then backup instance won't be able to delete all
    images exceeding rotation limit causing api failure.

    This patch handles ImageNotFound exception during deleting backup
    images, logs a warning message and continues deleting all of the
    remaining images.

    Closes-Bug: #1634773
    Change-Id: Ib999138ddbbcd62e0c99a69e5c6d37b4e49f871f
    (cherry picked from commit cc5d3437cabab4688df8492e071ea8c597311113)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.5

This issue was fixed in the openstack/nova 14.0.5 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.