Revert resize does not delete instance directory with Ceph

Bug #1631481 reported by Feodor Tersin on 2016-10-07
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Feodor Tersin

Bug Description

Resize revertion leaves instance directory on the second host with Ceph image backend. As the result the second attempt to resize the instance to the same host fails with n-cpu.log:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/", line 3942, in finish_resize
    disk_info, image_meta)
  File "/opt/stack/nova/nova/compute/", line 3907, in _finish_resize
  File "/usr/lib/python2.7/site-packages/oslo_utils/", line 220, in __exit__
  File "/usr/lib/python2.7/site-packages/oslo_utils/", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/", line 3902, in _finish_resize
    block_device_info, power_on)
  File "/opt/stack/nova/nova/virt/libvirt/", line 7185, in finish_migration
  File "/opt/stack/nova/nova/virt/libvirt/", line 2845, in _ensure_console_log_for_instance
    libvirt_utils.file_open(console_file, 'a').close()
  File "/opt/stack/nova/nova/virt/libvirt/", line 313, in file_open
    return open(*args, **kwargs)
IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log'

$ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/
total 24
-rw-rw-r--. 1 qemu qemu 19342 Oct 7 21:23 console.log
-rw-rw-r--. 1 stack libvirtd 2762 Oct 7 21:22 libvirt.xml

Steps to reproduce:
1 Run 2-nodes devstack with Ceph image backend
2 Run an instance
 $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
3 Disable the instance host
 $ nova service-disable nova-compute
4 Resize the instance to another host
 $ nova migrate inst-1
5 Revert resize
 $ nova resize-revert inst-1
6 Resize the instance again
 $ nova migrate inst-1
7 Check the instance state

Actual result - the instance is in error state.
Expected result - the instance is in verify_resize state.

Check n-cpu.log on the second node, where the instance was migrated.

This has been reproduced on master
commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015
Merge: 870a77f 453e71d
Author: Jenkins <email address hidden>
Date: Wed Oct 5 01:35:48 2016 +0000

Feodor Tersin (ftersin) on 2016-10-07
description: updated
Feodor Tersin (ftersin) wrote :

compute manater revert_resize calls driver.destroy on a target host with destroy_disks=False (due to shared block storage flag of Ceph image backend), because that driver.destroy does not instance directory. driver.destroy could delete it with appropriate migrate_data, but no such data exists for revert_resize operation.

Feodor Tersin (ftersin) on 2016-10-12
Changed in nova:
assignee: nobody → Feodor Tersin (ftersin)

Related fix proposed to branch: master

Changed in nova:
status: New → In Progress

Fix proposed to branch: master

Change abandoned by Feodor Tersin (<email address hidden>) on branch: master
Reason: in favor of I8d65c904

Change abandoned by Feodor Tersin (<email address hidden>) on branch: master
Reason: in favor of I8d65c904 and its probably descendants (to delete not used migrate_data param)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the review queue sane. Please feel free to restore the change if you're still working on it.

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers