Resize/migrate cannot reschedule

[hougangliu]

In nova/compute/manager.py: def prep_resize()

The design intends to catch every host error exception and then reschedule for resize/migrate like below:

        with self._error_out_instance_on_exception(context, instance['uuid'],
                                                   reservations):
            self.conductor_api.notify_usage_exists(
                    context, instance, current_period=True)
            self._notify_about_instance_usage(
                    context, instance, "resize.prep.start")
            try:
                self._prep_resize(context, image, instance,
                                  instance_type, reservations,
                                  request_spec, filter_properties,
                                  node)
            except Exception: <<<<<intend to catch host exception, and reschedule for resize/migrate
                # try to re-schedule the resize elsewhere:
                exc_info = sys.exc_info()
                self._reschedule_resize_or_reraise(context, image, instance,
                        exc_info, instance_type, reservations, request_spec,
                        filter_properties)

However, in self._prep_resize(), it would cast request of 'resize_instance()'. Thus, self._prep_resize() would return before resize_instance() finished. And resize_instance() may throw exception and reschedule is in need, but the exception would not be caught by prep_resize() and reschedule would not work.

[Guangya Liu (Jay Lau) (jay-lau-513)]

Perhaps we can add some retry logic in self._prep_resize

[Sahid Orentino (sahid-ferdjaoui)]

We need to wait for the response, if an exception occurs. What about using call instead of cast in nova/compute/rpcapi.py ?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/64284

[Sylvain Bauza (sylvain-bauza)]

Have to check the trunk to check if that bug has been fixed. Some paches came in about the subject, so that needs to be verified.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172044

[OpenStack Infra (hudson-openstack)]

Change abandoned by sahid (<email address hidden>) on branch: master
Review: https://review.openstack.org/172044

[melanie witt (melwitt)]

Resetting to New as the proposed patch was abandoned and this bug is also old and needs to be reverified

[Alexis Lee (alexisl)]

@melwitt: setting to Confirmed as sahid and sdague have previously confirmed this bug

[Alexis Lee (alexisl)]

OK after conferring with markus_z, this is an old bug so setting to Incomplete.

@hougangliu please can you try to reproduce? @sahid, @sdague also free to try.

[Alexis Lee (alexisl)]

This seems like a Tasks thing to me, specifically johnthetubaguy says:

"hmm, maybe, its the orchestration piece we want there I guess. The
original plan was to first do the move to conductor work for resize,
migrate and live migrate, the inter-compute stuff should be driven
centrally by the conductor, then we can wrap more structured handling
around that. If I am remembering correctly (I had a blueprint on this I
never had time to complete)"

So the long and short is, fixing this bug will require a lot of work (still worth trying) and a lot of care to fit in with project plans. We invented tag requires-large-refactor to mark this.

[Sean Dague (sdague)]

This is a bigger feature / structural change. And needs to come in through the specs process.