Revert resize does not delete instance directory with Ceph

Bug #1631481 reported by Feodor Tersin on 2016-10-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Feodor Tersin

Bug Description

Resize revertion leaves instance directory on the second host with Ceph image backend. As the result the second attempt to resize the instance to the same host fails with n-cpu.log:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 3942, in finish_resize
    disk_info, image_meta)
  File "/opt/stack/nova/nova/compute/manager.py", line 3907, in _finish_resize
    old_instance_type)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 3902, in _finish_resize
    block_device_info, power_on)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7185, in finish_migration
    self._ensure_console_log_for_instance(instance)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2845, in _ensure_console_log_for_instance
    libvirt_utils.file_open(console_file, 'a').close()
  File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 313, in file_open
    return open(*args, **kwargs)
IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/console.log'

$ ll ~/data/nova/instances/ad52ca5b-bb65-4f7c-87e8-750cb3cd9c5e/
total 24
-rw-rw-r--. 1 qemu qemu 19342 Oct 7 21:23 console.log
-rw-rw-r--. 1 stack libvirtd 2762 Oct 7 21:22 libvirt.xml

Steps to reproduce:
1 Run 2-nodes devstack with Ceph image backend
2 Run an instance
 $ nova boot --image cirros-0.3.4-x86_64-disk --flavor t1.nano inst-1
3 Disable the instance host
 $ nova service-disable 172.16.1.10 nova-compute
4 Resize the instance to another host
 $ nova migrate inst-1
5 Revert resize
 $ nova resize-revert inst-1
6 Resize the instance again
 $ nova migrate inst-1
7 Check the instance state

Actual result - the instance is in error state.
Expected result - the instance is in verify_resize state.

Check n-cpu.log on the second node, where the instance was migrated.

This has been reproduced on master
commit 9c89e07d17b5eb441682e3b8fad8b270f37f7015
Merge: 870a77f 453e71d
Author: Jenkins <email address hidden>
Date: Wed Oct 5 01:35:48 2016 +0000

Feodor Tersin (ftersin) on 2016-10-07
description: updated
Feodor Tersin (ftersin) wrote :

compute manater revert_resize calls driver.destroy on a target host with destroy_disks=False (due to shared block storage flag of Ceph image backend), because that driver.destroy does not instance directory. driver.destroy could delete it with appropriate migrate_data, but no such data exists for revert_resize operation.

Feodor Tersin (ftersin) on 2016-10-12
Changed in nova:
assignee: nobody → Feodor Tersin (ftersin)

Related fix proposed to branch: master
Review: https://review.openstack.org/385492

Changed in nova:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/389593

Change abandoned by Feodor Tersin (<email address hidden>) on branch: master
Review: https://review.openstack.org/385493
Reason: in favor of I8d65c904

Change abandoned by Feodor Tersin (<email address hidden>) on branch: master
Review: https://review.openstack.org/385492
Reason: in favor of I8d65c904 and its probably descendants (to delete not used migrate_data param)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/389593
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the review queue sane. Please feel free to restore the change if you're still working on it.

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.openstack.org/389593

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers