VolumeDetachTask loop

[Kevin Fox]

If the instance is deleted before heat detaches a volume, but the volume still exists, the stack seems to get stuck ignoring the 404. I see the following constantly streaming into the logs:

2013-11-14 12:47:18.419 8734 DEBUG heat.engine.scheduler [-] Task VolumeDetachTask(961852f2-02db-4f10-af6e-8737d0a98bd7 -/> 73f4be6d-a41e-4721-8cb6-4565b84cb97d) sleeping _sleep /usr/lib/python2.6/site-packages/heat/engine/scheduler.py:108
2013-11-14 12:47:18.908 8734 DEBUG heat.engine.scheduler [-] Task VolumeDetachTask(04dfd652-4d85-4609-9116-5af802bfa5f6 -/> d2ba9dd9-308a-4fec-88ae-114834765f5f) running step /usr/lib/python2.6/site-packages/heat/engine/scheduler.py:164
2013-11-14 12:47:18.963 8734 DEBUG urllib3.connectionpool [-] "DELETE /v2/a54290d67d044138b855cd0cdf9b737c/servers/d2ba9dd9-308a-4fec-88ae-114834765f5f/os-volume_attachments/04dfd652-4d85-4609-9116-5af802bfa5f6 HTTP/1.1" 404 78 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295

[Kevin Fox (kevin-fox-y)]

Note, this is with a DELETE_IN_PROGRESS stack.

[Kevin Fox (kevin-fox-y)]

The delete eventually failed due to timeout, and reissuing the delete caused the stack to fully delete.

[Kevin Fox (kevin-fox-y)]

A little more background, I was having some difficulty with the cinder volume and deleted the instance to see if I could get cinder to let go.

[Steve Baker (steve-stevebaker)]

Failing on timeout isn't the worst failure mode, but we should fix this so the stack deletes immediately.

[Clint Byrum (clint-fewbar)]

Has workaround, but it is ugly. We should handle problems like this gracefully.

[Pavlo Shchelokovskyy (pshchelo)]

Kevin, could you try if https://review.openstack.org/#/c/72681 fixes this problem?

[Pavlo Shchelokovskyy (pshchelo)]

I've tried it myself and seems like https://review.openstack.org/#/c/72681 fixes this bug

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72681
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d2b9b90777abe073b59cd8a4a7279f7dc3e18d00
Submitter: Jenkins
Branch: master

commit d2b9b90777abe073b59cd8a4a7279f7dc3e18d00
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Mon Feb 3 16:04:06 2014 +0200

    Add handle_update to VolumeAttachment

    Updates for volume attachments were handled in UpdateReplace manner,
    which fails when in need to attach new volume to the same mount
    (or attach the same volume to different mount on the same server,
    or attaching the same volume to different server).

    This patch makes all properties of VolumeAttachment
    to be update_allowed, adds necessary handle_update logic
    and fixes VolumeDetachTask to reliably detach old volume
    so that new one can be immediately attached.

    Closes-Bug: #1269686
    Closes-Bug: #1251439
    Change-Id: I0aee64036bf8125f76e7f427716412a326ff5eef

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/85905

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86461

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86461
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=db9545ea9ee8770ae693b0920e6211a74a056a20
Submitter: Jenkins
Branch: milestone-proposed

commit db9545ea9ee8770ae693b0920e6211a74a056a20
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Mon Feb 3 16:04:06 2014 +0200

    Add handle_update to VolumeAttachment

    Updates for volume attachments were handled in UpdateReplace manner,
    which fails when in need to attach new volume to the same mount
    (or attach the same volume to different mount on the same server,
    or attaching the same volume to different server).

    This patch makes all properties of VolumeAttachment
    to be update_allowed, adds necessary handle_update logic
    and fixes VolumeDetachTask to reliably detach old volume
    so that new one can be immediately attached.

    Change-Id: Id81a53a34b5bb2dfbc9425b91be0d02539fe2456
    Closes-Bug: #1269686
    Closes-Bug: #1251439