[heat] Cancel update action for stack is failed.

Bug #1570825 reported by Yury Tregubov
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
In Progress
High
Sergey Kraynev

Bug Description

The following test:

Test check the possibility to cancel update
      1. Create new stack
      2. Launch heat stack-update stack_name
      3. Launch heat stack-cancel-update stack_name while update
      operation is in progress
      4. Check state of stack after cancel update

Is failed on step 3 with error ROLLBACK_FAILED.

Stack is created with following template:

description: 'Template which creates single instance
 '
heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
  port:
   type: OS::Neutron::Port
   properties:
     network: {get_param: network}
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: port}
   type: OS::Nova::Server

During update the instance type is changed from tiny to small https://github.com/Mirantis/mos-integration-tests/blob/master/mos_tests/heat/heat_test.py#L719

The problem was seen first time on 9.0 mitaka iso #197 and is seen on iso #201 was well at the moment.

Dina Belova (dbelova)
tags: added: area-heat
Changed in mos:
assignee: nobody → MOS Heat (mos-heat)
importance: Undecided → High
milestone: none → 9.0
status: New → Confirmed
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

incomplete on fuel-9.0-223-2016-04-20_02-00-00.iso

Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

Sorry - invalid

Changed in mos:
status: Confirmed → Invalid
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

After research, I was able to reproduce this bug

Changed in mos:
status: Invalid → Confirmed
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Fix is on review in upstream https://review.openstack.org/#/c/306490/

We need to backport the fix to MOS 9.0 and MOS 10.0.

Changed in mos:
assignee: MOS Heat (mos-heat) → Oleksii Chuprykov (ochuprykov)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (stable/mitaka)

Change abandoned by Sergey Kraynev <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/20991
Reason: wrong branch

Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/20997
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 95493e5e431275b259b4a8d8b137c02bdb388d4c
Author: Oleksii Chuprykov <email address hidden>
Date: Fri May 20 14:23:00 2016

Fix cancel update for nova server with defined port

This particular patch fixes a behaviour of cancel update for nova
server with defined port, so there are no ports manageable by nova.
We have these issues while restoring ports after rollback:
1) We doesn't detach any ports from current server, because we
doesn't save them to resoruce data. (we store this data after
succesfull create of the server)
2) Detaching an interface from current server will fail, if the
server will be in building state, so we need to wait until server
will be in active or in error state.
Refresh ports list to solve problem (1).
Wait until nova moves to active/error state to solve (2).

Closes-Bug: #1570825
Partial-Bug: #1570908
Change-Id: I737736f572aae4d9ac67b8878bc40ca6b3ea4674

Changed in mos:
status: In Progress → Fix Committed
tags: added: on-verification
Revision history for this message
Evgeny Sikachev (esikachev) wrote :

verified on 429 iso, mos 9.0

Changed in mos:
status: Fix Committed → Fix Released
tags: removed: on-verification
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

The error can be found on the configuration 1 compute-1 controller
For find bug need run this test a lot of times

Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :

Couldn't fix it in upstream, because we need to merge another fix for cancel update in convergence mode: https://review.openstack.org/#/c/279406/ .

Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :
Changed in mos:
status: Fix Released → Confirmed
Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :

We should implement cancel update in convergence before fixing it. Partial fix for only legacy is not OK, because convergence is turned on by default.

Revision history for this message
Oleksii Chuprykov (ochuprykov) wrote :

This is actually nova bug https://bugs.launchpad.net/mos/+bug/1581037 Unfortunately, it marked as Won't Fix. So it was an attempt to find workaround in Heat, but it fixes the bug only partially. Sometimes cancel update is failing. I will ask nova team about status of this bug, probably they will fix it, or we will try to find new workaround or move this bug to Won't Fix.

Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

We have some new fix for this error
According to my research is to solve the problem
https://review.fuel-infra.org/#/c/25863/

Changed in mos:
status: Confirmed → Fix Committed
Changed in mos:
milestone: 9.0 → 9.1
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

on verification

tags: added: on-verification
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (10.0/newton)

Fix proposed to branch: 10.0/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/27663

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Fix was backported for 10.0

Revision history for this message
Sergey Sokolov (ssokolov-d) wrote :

The problem is also reproduced in MK24, see http://paste.openstack.org/show/593415/
heat versions:
heat-api 1:6.1.0-4~u14.04+mos4
heat-api-cfn 1:6.1.0-4~u14.04+mos4
heat-api-cloudwatch 1:6.1.0-4~u14.04+mos4
heat-common 1:6.1.0-4~u14.04+mos4
heat-engine 1:6.1.0-4~u14.04+mos4
python-heat 1:6.1.0-4~u14.04+mos4
python-heatclient 1.1.0-2~u14.04+mos2

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Sergey, so it means, that it's MOS 6.1. I will add this version to list of affected versions.

Revision history for this message
Sergey Kraynev (skraynev) wrote :

After researching I found, that it does not related with current bug, because 6.1 has the another root cause. So steps for reproducing are the same, but fix should be different

Revision history for this message
Sergey Kraynev (skraynev) wrote :

After researching, I figured out, that the bug was reported for deployment, which uses Mitaka branch.

So fix is already here.
The root cause it, that base example of the bug tell use network directly for creation Server.
It's valid case, but it does not support rollback and cancel commands, because nova by design delete ports created for server by nova.
So there is one fix for it - is to create port by using network parameter and then use this port for creation Server resource.

I will try to prepare update for doc with description this issue.

description: updated
no longer affects: mos/6.1.x
Revision history for this message
Sergey Kraynev (skraynev) wrote :

Template, which should not be used:

heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: network} - this option does not supported for rollback and cancel update for replace. Port should be used instead.
   type: OS::Nova::Server

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/heat (9.0/mitaka)

Related fix proposed to branch: 9.0/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29628

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/heat (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29680

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (stable/mitaka)

Change abandoned by Sergey Kraynev <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/29680

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to openstack/heat (9.0/mitaka)

Related fix proposed to branch: 9.0/mitaka
Change author: Sergey Kraynev <email address hidden>
Review: https://review.fuel-infra.org/29681

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (9.0/mitaka)

Change abandoned by Sergey Kraynev <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/29628
Reason: in favor of the https://review.fuel-infra.org/#/c/29681/1

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (mcp/newton)

Fix proposed to branch: mcp/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33420

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote :

Fix proposed to branch: mcp/newton
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33459

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (11.0/ocata)

Fix proposed to branch: 11.0/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/33885

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote :

Fix proposed to branch: 11.0/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34460

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (mcp/ocata)

Fix proposed to branch: mcp/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34527

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote :

Fix proposed to branch: mcp/ocata
Change author: Oleksii Chuprykov <email address hidden>
Review: https://review.fuel-infra.org/34754

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (mcp/newton)

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33420
Reason: is available in upstream since newton

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (mcp/ocata)

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34527
Reason: is available in upstream since newton

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (11.0/ocata)

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33885
Reason: is available in upstream since newton

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote :

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34460
Reason: 11.0/ocata branch is deprecated in favor of mcp/ocata

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (mcp/ocata)

Reviewed: https://review.fuel-infra.org/34754
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 18debeb884ea825b4d33cae1e3c63c566a34fc7f
Author: Oleksii Chuprykov <email address hidden>
Date: Tue Apr 25 18:54:47 2017

Fix races in cancel update

There is issue in nova with detaching port from a server.
Sometimes nova can delete a port instead of detach it. This situation
happens if we ask nova to detach port from a server that was moved
to active state right before detach was called. So, we can wait some
time before detach the port. This hopefully fix the problem in the most
situations.

Change-Id: Ib1b5d0f713870d66e66d07e949e89cd039d87936
Closes-Bug: #1570825

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (mcp/newton)

Reviewed: https://review.fuel-infra.org/33459
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 3ea09ca7ef13f4301f4f5076000ea159cfffb656
Author: Oleksii Chuprykov <email address hidden>
Date: Thu Apr 27 11:16:39 2017

Fix races in cancel update

There is issue in nova with detaching port from a server.
Sometimes nova can delete a port instead of detach it. This situation
happens if we ask nova to detach port from a server that was moved
to active state right before detach was called. So, we can wait some
time before detach the port. This hopefully fix the problem in the most
situations.

Change-Id: Ib1b5d0f713870d66e66d07e949e89cd039d87936
Closes-Bug: #1570825

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (10.0/newton)

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: 10.0/newton
Review: https://review.fuel-infra.org/27663
Reason: 10.0/newton branch is deprecated, merged into mcp/newton branch instead

https://review.fuel-infra.org/#/c/33459/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers