[heat] Cancel update action for stack is failed.

[Yury Tregubov]

The following test:

Test check the possibility to cancel update
      1. Create new stack
      2. Launch heat stack-update stack_name
      3. Launch heat stack-cancel-update stack_name while update
      operation is in progress
      4. Check state of stack after cancel update

Is failed on step 3 with error ROLLBACK_FAILED.

Stack is created with following template:

description: 'Template which creates single instance
 '
heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
  port:
   type: OS::Neutron::Port
   properties:
     network: {get_param: network}
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: port}
   type: OS::Nova::Server

During update the instance type is changed from tiny to small https://github.com/Mirantis/mos-integration-tests/blob/master/mos_tests/heat/heat_test.py#L719

The problem was seen first time on 9.0 mitaka iso #197 and is seen on iso #201 was well at the moment.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Oleksii Chuprykov (ochuprykov)]

Link to upstream bug: https://bugs.launchpad.net/heat/+bug/1570908

[Alexander Nagovitsyn (gluk12189)]

incomplete on fuel-9.0-223-2016-04-20_02-00-00.iso

[Alexander Nagovitsyn (gluk12189)]

Sorry - invalid

[Alexander Nagovitsyn (gluk12189)]

After research, I was able to reproduce this bug

[Alexander Nagovitsyn (gluk12189)]

Add on review
https://review.openstack.org/#/c/306490/

[Timur Nurlygayanov (tnurlygayanov)]

Fix is on review in upstream https://review.openstack.org/#/c/306490/

We need to backport the fix to MOS 9.0 and MOS 10.0.

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Kraynev <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/20991
Reason: wrong branch

[Oleksii Chuprykov (ochuprykov)]

Fix: https://review.fuel-infra.org/#/c/20997/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20997
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 95493e5e431275b259b4a8d8b137c02bdb388d4c
Author: Oleksii Chuprykov <email address hidden>
Date: Fri May 20 14:23:00 2016

Fix cancel update for nova server with defined port

This particular patch fixes a behaviour of cancel update for nova
server with defined port, so there are no ports manageable by nova.
We have these issues while restoring ports after rollback:
1) We doesn't detach any ports from current server, because we
doesn't save them to resoruce data. (we store this data after
succesfull create of the server)
2) Detaching an interface from current server will fail, if the
server will be in building state, so we need to wait until server
will be in active or in error state.
Refresh ports list to solve problem (1).
Wait until nova moves to active/error state to solve (2).

Closes-Bug: #1570825
Partial-Bug: #1570908
Change-Id: I737736f572aae4d9ac67b8878bc40ca6b3ea4674

[Evgeny Sikachev (esikachev)]

verified on 429 iso, mos 9.0

[Alexander Nagovitsyn (gluk12189)]

The error can be found on the configuration 1 compute-1 controller
For find bug need run this test a lot of times

[Oleksii Chuprykov (ochuprykov)]

Couldn't fix it in upstream, because we need to merge another fix for cancel update in convergence mode: https://review.openstack.org/#/c/279406/ .

[Oleksii Chuprykov (ochuprykov)]

Blocked by this patch https://review.openstack.org/#/c/301483/12

[Oleksii Chuprykov (ochuprykov)]

We should implement cancel update in convergence before fixing it. Partial fix for only legacy is not OK, because convergence is turned on by default.

[Oleksii Chuprykov (ochuprykov)]

This is actually nova bug https://bugs.launchpad.net/mos/+bug/1581037 Unfortunately, it marked as Won't Fix. So it was an attempt to find workaround in Heat, but it fixes the bug only partially. Sometimes cancel update is failing. I will ask nova team about status of this bug, probably they will fix it, or we will try to find new workaround or move this bug to Won't Fix.

[Alexander Nagovitsyn (gluk12189)]

We have some new fix for this error
According to my research is to solve the problem
https://review.fuel-infra.org/#/c/25863/

[Alexander Nagovitsyn (gluk12189)]

on verification

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 10.0/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/27663

[Sergey Kraynev (skraynev)]

Fix was backported for 10.0

[Sergey Sokolov (ssokolov-d)]

The problem is also reproduced in MK24, see http://paste.openstack.org/show/593415/
heat versions:
heat-api 1:6.1.0-4~u14.04+mos4
heat-api-cfn 1:6.1.0-4~u14.04+mos4
heat-api-cloudwatch 1:6.1.0-4~u14.04+mos4
heat-common 1:6.1.0-4~u14.04+mos4
heat-engine 1:6.1.0-4~u14.04+mos4
python-heat 1:6.1.0-4~u14.04+mos4
python-heatclient 1.1.0-2~u14.04+mos2

[Sergey Kraynev (skraynev)]

Sergey, so it means, that it's MOS 6.1. I will add this version to list of affected versions.

[Sergey Kraynev (skraynev)]

After researching I found, that it does not related with current bug, because 6.1 has the another root cause. So steps for reproducing are the same, but fix should be different

[Sergey Kraynev (skraynev)]

After researching, I figured out, that the bug was reported for deployment, which uses Mitaka branch.

So fix is already here.
The root cause it, that base example of the bug tell use network directly for creation Server.
It's valid case, but it does not support rollback and cancel commands, because nova by design delete ports created for server by nova.
So there is one fix for it - is to create port by using network parameter and then use this port for creation Server resource.

I will try to prepare update for doc with description this issue.

[Sergey Kraynev (skraynev)]

Template, which should not be used:

heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: network} - this option does not supported for rollback and cancel update for replace. Port should be used instead.
   type: OS::Nova::Server

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29628

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29680

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Kraynev <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/29680

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29681

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Kraynev <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/29628
Reason: in favor of the https://review.fuel-infra.org/#/c/29681/1

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33420

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33459

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 11.0/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33885

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 11.0/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34460

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34527

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34754

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33420
Reason: is available in upstream since newton

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34527
Reason: is available in upstream since newton

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33885
Reason: is available in upstream since newton

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34460
Reason: 11.0/ocata branch is deprecated in favor of mcp/ocata

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/34754
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 18debeb884ea825b4d33cae1e3c63c566a34fc7f
Author: Oleksii Chuprykov <email address hidden>
Date: Tue Apr 25 18:54:47 2017

Fix races in cancel update

There is issue in nova with detaching port from a server.
Sometimes nova can delete a port instead of detach it. This situation
happens if we ask nova to detach port from a server that was moved
to active state right before detach was called. So, we can wait some
time before detach the port. This hopefully fix the problem in the most
situations.

Change-Id: Ib1b5d0f713870d66e66d07e949e89cd039d87936
Closes-Bug: #1570825

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/33459
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 3ea09ca7ef13f4301f4f5076000ea159cfffb656
Author: Oleksii Chuprykov <email address hidden>
Date: Thu Apr 27 11:16:39 2017

Fix races in cancel update

There is issue in nova with detaching port from a server.
Sometimes nova can delete a port instead of detach it. This situation
happens if we ask nova to detach port from a server that was moved
to active state right before detach was called. So, we can wait some
time before detach the port. This hopefully fix the problem in the most
situations.

Change-Id: Ib1b5d0f713870d66e66d07e949e89cd039d87936
Closes-Bug: #1570825

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 10.0/newton
Review: https://review.fuel-infra.org/27663
Reason: 10.0/newton branch is deprecated, merged into mcp/newton branch instead

https://review.fuel-infra.org/#/c/33459/