Rally

HeatStacks are not properly deleted

Bug #1450997 reported by Boris Pavlovic
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Rally
Fix Released
High
Kairat Kushaev
Rally 0.0.4

Bug Description

2015-05-01 15:24:11.650 | 2015-05-01 15:24:11.610 | {
2015-05-01 15:24:11.651 | 2015-05-01 15:24:11.612 | "kw": {
2015-05-01 15:24:11.653 | 2015-05-01 15:24:11.614 | "runner": {
2015-05-01 15:24:11.655 | 2015-05-01 15:24:11.616 | "type": "constant",
2015-05-01 15:24:11.686 | 2015-05-01 15:24:11.617 | "concurrency": 3,
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.619 | "times": 6
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.621 | },
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.624 | "sla": {
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.626 | "failure_rate": {
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.627 | "max": 0
2015-05-01 15:24:11.687 | 2015-05-01 15:24:11.630 | }
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.632 | },
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.633 | "context": {
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.635 | "users": {
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.637 | "users_per_tenant": 3,
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.639 | "tenants": 2
2015-05-01 15:24:11.688 | 2015-05-01 15:24:11.641 | },
2015-05-01 15:24:11.689 | 2015-05-01 15:24:11.643 | "stacks": {
2015-05-01 15:24:11.689 | 2015-05-01 15:24:11.644 | "resources_per_stack": 10,
2015-05-01 15:24:11.689 | 2015-05-01 15:24:11.646 | "stacks_per_tenant": 2
2015-05-01 15:24:11.689 | 2015-05-01 15:24:11.648 | }
2015-05-01 15:24:11.689 | 2015-05-01 15:24:11.650 | }
2015-05-01 15:24:11.691 | 2015-05-01 15:24:11.651 | },
2015-05-01 15:24:11.693 | 2015-05-01 15:24:11.653 | "name": "HeatStacks.list_stacks_and_events",
2015-05-01 15:24:11.694 | 2015-05-01 15:24:11.655 | "pos": 0
2015-05-01 15:24:11.698 | 2015-05-01 15:24:11.657 | }
2015-05-01 15:24:11.699 | 2015-05-01 15:24:11.659 | 2015-05-01 15:24:11.607 3333 INFO rally.benchmark.context.users [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | Starting: Enter context: `users`
2015-05-01 15:24:12.884 | 2015-05-01 15:24:12.845 | 2015-05-01 15:24:12.843 3333 INFO rally.benchmark.context.users [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | Completed: Enter context: `users`
2015-05-01 15:24:12.886 | 2015-05-01 15:24:12.846 | 2015-05-01 15:24:12.843 3333 INFO rally.benchmark.context.stacks [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | Starting: Enter context: `Stacks`
2015-05-01 15:24:26.364 | 2015-05-01 15:24:26.324 | 2015-05-01 15:24:26.322 3333 INFO rally.benchmark.context.stacks [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | Completed: Enter context: `Stacks`
2015-05-01 15:24:26.372 | 2015-05-01 15:24:26.332 | 2015-05-01 15:24:26.330 26615 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 0 START
2015-05-01 15:24:26.375 | 2015-05-01 15:24:26.335 | 2015-05-01 15:24:26.333 26617 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 1 START
2015-05-01 15:24:26.379 | 2015-05-01 15:24:26.340 | 2015-05-01 15:24:26.337 26619 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 2 START
2015-05-01 15:24:27.533 | 2015-05-01 15:24:27.494 | 2015-05-01 15:24:27.492 26617 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 1 END: OK
2015-05-01 15:24:27.541 | 2015-05-01 15:24:27.500 | 2015-05-01 15:24:27.498 26617 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 3 START
2015-05-01 15:24:27.856 | 2015-05-01 15:24:27.815 | 2015-05-01 15:24:27.813 26619 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 2 END: OK
2015-05-01 15:24:27.857 | 2015-05-01 15:24:27.817 | 2015-05-01 15:24:27.816 26619 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 4 START
2015-05-01 15:24:28.227 | 2015-05-01 15:24:28.188 | 2015-05-01 15:24:28.186 26615 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 0 END: OK
2015-05-01 15:24:28.230 | 2015-05-01 15:24:28.190 | 2015-05-01 15:24:28.188 26615 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 5 START
2015-05-01 15:24:29.175 | 2015-05-01 15:24:29.136 | 2015-05-01 15:24:29.134 26617 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 3 END: OK
2015-05-01 15:24:29.372 | 2015-05-01 15:24:29.333 | 2015-05-01 15:24:29.331 26619 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 4 END: OK
2015-05-01 15:24:29.809 | 2015-05-01 15:24:29.770 | 2015-05-01 15:24:29.768 26615 INFO rally.benchmark.runners.base [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | ITER: 5 END: OK
2015-05-01 15:24:29.818 | 2015-05-01 15:24:29.779 | 2015-05-01 15:24:29.775 3333 INFO rally.benchmark.context.stacks [-] Task 81f8f2b8-84f4-4ecb-94af-de05183d4125 | Starting: Exit context: `Stacks`
2015-05-01 15:24:36.635 | 2015-05-01 15:24:36.595 | 2015-05-01 15:24:36.593 3333 WARNING rally.benchmark.context.cleanup.manager [-] Resource deletion failed, max retries exceeded for heat.stacks: 1d6941ee-f156-43f7-9829-6a26eda942ed. Reason: ERROR: The Stack (thintybroy) could not be found.
2015-05-01 15:24:36.640 | 2015-05-01 15:24:36.601 | 2015-05-01 15:24:36.599 3333 WARNING rally.benchmark.context.cleanup.manager [-] Resource deletion failed, max retries exceeded for heat.stacks: b22e682f-1e9e-4936-8a9f-971a9f7a2612. Reason: ERROR: The Stack (fanbxabdtd) could not be found.

Boris Pavlovic (boris-42)
Changed in rally:
status: New → Confirmed
status: Confirmed → In Progress
status: In Progress → Triaged
importance: Undecided → High
Kairat Kushaev (kkushaev)
Changed in rally:
assignee: nobody → Kairat Kushaev (kkushaev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to rally (master)

Fix proposed to branch: master
Review: https://review.openstack.org/180068

Changed in rally:
status: Triaged → In Progress
Boris Pavlovic (boris-42)
Changed in rally:
milestone: none → 0.0.4
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to rally (master)

Reviewed: https://review.openstack.org/180068
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=4e4a1c6eef7adbb2f1b7962a22033048897c552c
Submitter: Jenkins
Branch: master

commit 4e4a1c6eef7adbb2f1b7962a22033048897c552c
Author: kairat_kushaev <email address hidden>
Date: Tue May 5 11:58:02 2015 +0300

    Cleanup heat stacks 1 time per tenant

    Heat stacks are tenant-based, it means that when manager is
    returning list of stack for the whole tenant(not for user).
    So when rally is cleaning up stacks it needs to delete it
    only 1 time per tenant. Otherwise, it is very likely to get
    "stack not found" errors in log.

    Change-Id: Id8b0f967e2a1b35c8de1abdeee40fc060ce2a0d2
    Closes-bug: #1450997

Changed in rally:
status: In Progress → Fix Committed
Boris Pavlovic (boris-42)
Changed in rally:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.