stack update failed when update image_id of an instance with volume attached

[Song Li]

reproduce steps:
a, deploy an instance with a volume attached --- this step works well.
b, update the stack with another image_id for the instance. --- then will update fail.
------------------------------------------------------------------------------------
used template:

parameters:
  instance_type:
    type: string
    description: Type of the instance to be created.
    default: m1.tiny
    constraints:
      - allowed_values: [m1.tiny, m1.small]
        description:
          Value must be one of 'm1.small', 'm1.tiny'.
  image_id:
    type: string
    description: ID of the image to use for the instance to be created.
  availability_zone:
    type: string
    description: The Availability Zone to launch the instance.
    default: zone1
  volume_size:
    type: number
    description: Size of the volume to be created.
    default: 1
    constraints:
      - range: { min: 1, max: 2 }
        description: must be between 1 and 1024 Gb.
  cinder_availability_zone:
    type: string
    description: The Availability Zone to launch the instance.
    default: cinder1

resources:
  nova_instance:
    type: OS::Nova::Server
    properties:
      availability_zone: { get_param: availability_zone }
      image: { get_param: image_id }
      flavor: { get_param: instance_type }
  cinder_volume:
    type: OS::Cinder::Volume
    properties:
      size: { get_param: volume_size }
      availability_zone: { get_param: cinder_availability_zone }
  volume_attachment:
    type: OS::Cinder::VolumeAttachment
    properties:
      volume_id: { get_resource: cinder_volume }
      instance_uuid: { get_resource: nova_instance }
      mountpoint: /dev/sdb
-----------------------------------------------------------------------------------
Error reported in logs:
2014-12-15 11:42:13.688 23713 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.1.0.113
2014-12-15 11:42:14.815 23713 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.1.0.113
2014-12-15 11:42:15.336 23713 INFO heat.engine.resource [-] UPDATE: CinderVolumeAttachment "volume_attachment" [798866a4-71ba-411e-a72c-17ea12066983] Stack "nova_cinder_attach" [433b25c3-b297-4381-a5a6-0003e8785980]
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource Traceback (most recent call last):
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 435, in _action_recorder
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource yield
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 690, in update
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource args=[after, tmpl_diff, prop_diff])
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/scheduler.py", line 303, in wrapper
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource step = next(subtask)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 479, in action_handler_task
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource while not check(handler_data):
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resources/volume.py", line 772, in check_update_complete
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource checker.start()
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/scheduler.py", line 190, in start
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource self.step()
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/scheduler.py", line 213, in step
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource next(self._runner)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/heat/engine/resources/volume.py", line 272, in __call__
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource device=self.device)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/v1_1/volumes.py", line 119, in create_server_volume
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource body, "volumeAttachment")
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 100, in _create
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource _resp, body = self.api.client.post(url, body=body)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 490, in post
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource return self._cs_request(url, 'POST', **kwargs)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 465, in _cs_request
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource resp, body = self._time_request(url, method, **kwargs)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 439, in _time_request
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource resp, body = self.request(url, method, **kwargs)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 433, in request
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource BadRequest: Invalid volume: status must be 'available' (HTTP 400) (Request-ID: req-179f7b0e-1b5c-4eb6-a324-5bf3d7ee4f59)
2014-12-15 11:42:15.336 23713 TRACE heat.engine.resource

[Song Li (lisong-cruise)]

debugged into CinderVolumeAttachment
    def handle_update(self, json_snippet, tmpl_diff, prop_diff):
        checkers = []
        if prop_diff:
            # Even though some combinations of changed properties
            # could be updated in UpdateReplace manner,
            # we still first detach the old resource so that
            # self.resource_id is not replaced prematurely
            volume_id = self.properties.get(self.VOLUME_ID)
            if self.VOLUME_ID in prop_diff:
                volume_id = prop_diff.get(self.VOLUME_ID)

            device = self.properties.get(self.DEVICE)
            if self.DEVICE in prop_diff:
                device = prop_diff.get(self.DEVICE)

            server_id = self.properties.get(self.INSTANCE_ID)
            print server_id
            detach_task = VolumeDetachTask(self.stack, server_id,
                                           self.resource_id)
            checkers.append(scheduler.TaskRunner(detach_task))

            if self.INSTANCE_ID in prop_diff:
                server_id = prop_diff.get(self.INSTANCE_ID)
            print server_id
            attach_task = VolumeAttachTask(self.stack, server_id,
                                           volume_id, device)

            checkers.append(scheduler.TaskRunner(attach_task))

        if checkers:
            checkers[0].start()
        return checkers
I print the old server id and current server id, both value is the new server id
6b6f87d5-4fa6-456d-94c9-3e9c7a857ee8
6b6f87d5-4fa6-456d-94c9-3e9c7a857ee8

[Song Li (lisong-cruise)]

From comment #1, I think the problem is that heat want to detach the volume first and attach it later.
But from the code, heat got the new server id rather than old server id, so the dettach action failed. Then it will fail when execute the attach action. As the volume is never dettached

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/142003

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/142003
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2c58f31063e58ebceab3a3aadf8ca0f8612b755a
Submitter: Jenkins
Branch: master

commit 2c58f31063e58ebceab3a3aadf8ca0f8612b755a
Author: Song Li <email address hidden>
Date: Tue Dec 16 14:49:28 2014 +0800

    Get correct server_id when udpate Volume Attachment

    Get the correct server id used for dettach the volume when
    updating CinderVolumeAttachment.

    Fixed bug #1402500

    Change-Id: I3eeb24e6eab8335475d6a76d80a461765089c9c0