run_instance thread may block if exception InstanceNotFound raised

[Zap Chang]

On compute node, before going into run_instance calling path, a lock named with instance’s UUID is acquired firstly.

If exception exception.InstanceNotFound is raised in following try statement, terminate_instance method which also acquires the same lock will be called in _shutdown_instance_even_if_deleted calling path. The greenthread is blocked since the lock isn’t released yet. This may cause instance state to be always pending.

def _run_instance(self, context, instance_uuid,
                      requested_networks=None,
                      injected_files=[],
                      admin_password=None,
                      is_first_time=False,
                      **kwargs):
        """Launch a new instance with specified options."""
        context = context.elevated()
        try:
            instance = self.db.instance_get_by_uuid(context, instance_uuid)
            self._check_instance_not_already_created(context, instance)
            image_meta = self._check_image_size(context, instance)
            self._start_building(context, instance)

            < code omitted ...>

        except exception.InstanceNotFound:
            LOG.exception(_("Instance not found."),
                          instance_uuid=instance_uuid)
            # assuming the instance was already deleted, run "delete" again
            # just in case
            self._shutdown_instance_even_if_deleted(context, instance_uuid)
            return
        except Exception as e:
            with utils.save_and_reraise_exception():
                self._set_instance_error_state(context, instance_uuid)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/6714

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6714
Committed: http://github.com/openstack/nova/commit/4bd4f756f107f94a264d8894203c00d10af322ee
Submitter: Jenkins
Branch: master

commit 4bd4f756f107f94a264d8894203c00d10af322ee
Author: MotoKen <email address hidden>
Date: Mon Apr 23 19:24:14 2012 +0800

    Removes unnecessary code in _run_instance.

    * Remove _is_instance_terminated and _shutdown_instance_even_if_deleted
      since run/terminate instance is synchronized.
    * Avoid potential deadlock.
    * Fixes bug 981316.

    Change-Id: I4fe7e2ea00b85d9e0802fe93655e67db3909e86b