Delete stack can get stuck due to uncaught exception

Bug #1518903 reported by Andrey Epifanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Sergey Kraynev
5.1.x
In Progress
High
MOS Maintenance
6.0.x
Fix Released
High
Sergey Kraynev
6.1.x
Fix Released
High
Sergey Kraynev

Bug Description

Backport of https://bugs.launchpad.net/heat/+bug/1386213

Customers often face this issue on production environments.
Pretty big stacks are not able to be deleted and consume a lot of resources.
Unfortunately we don't have a suitable workaround for this. Only manual modifying of BD helps here.

Changed in mos:
importance: Undecided → High
status: New → Confirmed
assignee: nobody → Sergey Kraynev (skraynev)
milestone: none → 7.0-updates
description: updated
Revision history for this message
Sergey Kraynev (skraynev) wrote :
Revision history for this message
Sergey Kraynev (skraynev) wrote :

Both patches are already merged for 7.0, so we do not need to port anything else to 7.0 branch

no longer affects: mos/7.0.x
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: tengqm <email address hidden>
Review: https://review.fuel-infra.org/14135

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/heat (openstack-ci/fuel-5.1/2014.1.1)

Change abandoned by Sergey Kraynev <email address hidden> on branch: openstack-ci/fuel-5.1/2014.1.1
Review: https://review.fuel-infra.org/13961
Reason: Abandoned in favor of https://review.fuel-infra.org/#/c/14135/

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-6.0-updates/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: tengqm <email address hidden>
Review: https://review.fuel-infra.org/14138

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-6.1/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: tengqm <email address hidden>
Review: https://review.fuel-infra.org/14139

Changed in mos:
milestone: 7.0-updates → 7.0-mu-2
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Retargeted to 6.1-mu-5 because of delayed reviews

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Invalid for 7.0-updates per feedback from Heat the fix is already in 7.0 branch

Changed in mos:
status: Confirmed → Invalid
milestone: 7.0-mu-2 → 7.0-updates
Revision history for this message
Roman Rufanov (rrufanov) wrote :

Customer found on MOS 5.1

tags: added: customer-found support
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (openstack-ci/fuel-6.0-updates/2014.2)

Reviewed: https://review.fuel-infra.org/14138
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 5ac6ca2be00cf1f17ff232fb678833d02144fc71
Author: tengqm <email address hidden>
Date: Mon Feb 15 13:44:50 2016

Prevent using long password for crypting trust

Check if encoded user credential may get truncated.

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit.

This patch checks if the encoded credential might be truncated when
it is generated and fail early during stack creation, or else it will
cause troubles later on when operating the stack.

Don't block stack deletion if user creds fail.

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit. When this incorrect behavior
happens silently, the user credential can not be retrieved.
In this case, stack deletion will abort due to uncaught exception.

This patch enables a stack deletion to handle this error so that stack
can be properly deleted.

Closes-Bug: #1518903
(cherry picked from commit 0d9af5c6833e26532bf798ebd0756e93d674b7fc)
(cherry picked from commit 9698da68040b0e3ac63c23e72f31d9739c76e03e)

Change-Id: Icc05b300542006c3306530949e9fa6726d76589c

Dmitry (dtsapikov)
tags: added: on-verification
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (openstack-ci/fuel-6.1/2014.2)

Reviewed: https://review.fuel-infra.org/14139
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: e073a2760b94b9da6d34a032d0b435102eb02702
Author: tengqm <email address hidden>
Date: Tue Dec 8 11:48:49 2015

Prevent using long password for crypting trust

Check if encoded user credential may get truncated

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit.

This patch checks if the encoded credential might be truncated when
it is generated and fail early during stack creation, or else it will
cause troubles later on when operating the stack.

Don't block stack deletion if user creds fail

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit. When this incorrect behavior
happens silently, the user credential can not be retrieved.
In this case, stack deletion will abort due to uncaught exception.

This patch enables a stack deletion to handle this error so that stack
can be properly deleted.

Closes-Bug: #1518903
(cherry picked from commit 0d9af5c6833e26532bf798ebd0756e93d674b7fc)
(cherry picked from commit 9698da68040b0e3ac63c23e72f31d9739c76e03e)

Change-Id: Icc05b300542006c3306530949e9fa6726d76589c

Revision history for this message
Dmitry (dtsapikov) wrote :

Bug was not reproduced before the patch was applied.
After updating I had exception that was added in the patch https://paste.mirantis.net/show/1915/

Revision history for this message
Dmitry (dtsapikov) wrote :

Bug was not reproduced on 6.1+mu5 before the patch was applied.
After updating I had exception that was added in the patch https://paste.mirantis.net/show/1915/

tags: removed: on-verification
Revision history for this message
Sergey Kraynev (skraynev) wrote :

Fix was proposed and just need to be tested / merged https://review.fuel-infra.org/#/c/14135/

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/14135
Submitter: Sergey Kraynev <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: c329b62eee7836c9c829895d9015725979e952d5
Author: tengqm <email address hidden>
Date: Wed Jul 6 11:24:58 2016

Prevent using long password for crypting trust

Check if encoded user credential may get truncated

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit.

This patch checks if the encoded credential might be truncated when
it is generated and fail early during stack creation, or else it will
cause troubles later on when operating the stack.

Don't block stack deletion if user creds fail

In some deployments, user passwords are generated by other
authentication infrastructures where the length of the Base64 encoded
credentials is longer than the DB limit. When this incorrect behavior
happens silently, the user credential can not be retrieved.
In this case, stack deletion will abort due to uncaught exception.

This patch enables a stack deletion to handle this error so that stack
can be properly deleted.

Closes-Bug: #1518903
(cherry picked from commit 0d9af5c6833e26532bf798ebd0756e93d674b7fc)
(cherry picked from commit 9698da68040b0e3ac63c23e72f31d9739c76e03e)

Change-Id: Icc05b300542006c3306530949e9fa6726d76589c

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.