revert a resize task and resize again or live migration will fail when use rbd

[赵明俊]

Description
===========
I resize an instance and revert it, it works correctly, but when I resize this instance again, the instance set to error, and I check the compute log like below.
I use the rbd backend, I think the reason should be like this, If instance use shared storage, revert resize will not delete the instance dir and files on destination host, this files owner is root, if resize the instance again and migrate to the same destination host, the IOError exception will be thrown.

Logs
==============
Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3950, in finish_resize
     disk_info, image_meta)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3915, in _finish_resize
     old_instance_type)
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
     self.force_reraise()
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3910, in _finish_resize
     block_device_info, power_on)
   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7223, in finish_migration
     self._ensure_console_log_for_instance(instance)
   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2867, in _ensure_console_log_for_instance
     libvirt_utils.file_open(console_file, 'a').close()
   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 313, in file_open
     return open(*args, **kwargs)
 IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/c1c4847d-0470-4fae-9060-6511a8e2c056/console.log'

[Matt Riedemann (mriedem)]

On which release is this happening? I believe the console.log ownership issue has been fixed in newer releases.

[Matt Riedemann (mriedem)]

See https://review.openstack.org/#/c/392643/ and https://review.openstack.org/#/c/466088/ - do you have those fixes?

[赵明俊 (falseuser)]

Thanks, I found this in Newton, those above commits can solve my problem, and why we do not directly delete the instance files and path on destination host after revert a resize task?

[赵明俊 (falseuser)]

It is need to delete the instance files and path on destination host after revert a resize task, I try a live migration after revert a resize task when use rbd, and I see the error log like below

Logs
=====================================
ERROR oslo_messaging.rpc.server [None req-77b787e3-f885-4783-a4bc-f60a1491d1cc admin admin] Exception during message handling: DestinationDiskExists: The supplied disk path (/opt/stack/data/nova/instances/44721008-5a73-43e6-8c91-3d31d88f93d2) already exists, it is expected not to exist.
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 77, in wrapped
ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 68, in wrapped
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 690, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function
ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 5362, in pre_live_migration
ERROR oslo_messaging.rpc.server migrate_data)
ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6791, in pre_...

[赵明俊 (falseuser)]

I found this in Pike (devstack-master), and do live migration again will succeed

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[Sean Dague (sdague)]

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/469814 in branch: master

[melanie witt (melwitt)]

Reading through the comments on this report, it sounds like there is no bug on the master branch and that this issue was noticed in the newton release, and thus we could close this bug and abandon the accompanying patch. Is that correct or have I misunderstood something?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Balazs Gibizer (<email address hidden>) on branch: master
Review: https://review.opendev.org/469814
Reason: Bug sits in incomplete status since 2017-10-31. Bug reporter did not clarified the question. Bug reported against Newton that is EOL.