tripleo

Forceful removal of containers in paunch/tripleo-container-manage leaves leftovers behind

Bug #1860004 reported by Bogdan Dobrelya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Bogdan Dobrelya
tripleo ussuri-2 "tripleo ussuri-2"

Bug Description

Paunch does docker/podman rm -f, when removes containers. It seems it
returns too early, having some leftovers (like docker service
endpoints) behind or pending for it to be removed later, in case of
big fat deamons.

Long story short, don't do rm -f and allow it to do its job gracefully
and without a hurry.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to paunch (master)

Fix proposed to branch: master
Review: https://review.opendev.org/702899

Changed in tripleo:
assignee: nobody → Bogdan Dobrelya (bogdando)
status: New → In Progress
Bogdan Dobrelya (bogdando)
Changed in tripleo:
importance: Undecided → High
milestone: none → ussuri-2
tags: added: queens-backport-potential
tags: added: train-backport-potential
Changed in tripleo:
status: In Progress → Triaged
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to paunch (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/702901

Bogdan Dobrelya (bogdando)
summary: - Forceful removal of containers in paunch leaves leftovers behind
+ Forceful removal of containers in paunch/tripleo-container-manage leaves
+ leftovers behind
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to paunch (master)

Reviewed: https://review.opendev.org/702899
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=3813fc7f2bd5768248f7c450d707a13c33f19fc3
Submitter: Zuul
Branch: master

commit 3813fc7f2bd5768248f7c450d707a13c33f19fc3
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 16 16:41:59 2020 +0100

    Do not force remove containers

    Paunch does docker/podman rm -f, when removes containers. It seems it
    returns too early, having some leftovers (like docker service
    endpoints) behind or pending for it to be removed later, in case of
    big fat deamons.

    Long story short, don't do rm -f and allow it to do its job gracefully
    and without a hurry.

    Change-Id: I346c49cb204f273bd7077ca5153412cda9846534
    Closes-Bug: #1860004
    Co-authored-by: Sergii Golovatiuk <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to paunch (stable/queens)

Reviewed: https://review.opendev.org/702901
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=d2a7acfa228a332acb9bb5b94f1f12cc045790bd
Submitter: Zuul
Branch: stable/queens

commit d2a7acfa228a332acb9bb5b94f1f12cc045790bd
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 16 16:41:59 2020 +0100

    Do not force remove containers

    Paunch does docker/podman rm -f, when removes containers. It seems it
    returns too early, having some leftovers (like docker service
    endpoints) behind or pending for it to be removed later, in case of
    big fat deamons.

    Long story short, don't do rm -f and allow it to do its job gracefully
    and without a hurry.

    Change-Id: I346c49cb204f273bd7077ca5153412cda9846534
    Closes-Bug: #1860004
    Co-authored-by: Sergii Golovatiuk <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit d39414e433c1a20e71dee4612ec1387e57620103)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to paunch (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/704668

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to paunch (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/704669

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to paunch (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/704672

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to paunch (stable/train)

Reviewed: https://review.opendev.org/704668
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=d56765a0aff2f8cf5761a26de86a0ec3a2e5bd0f
Submitter: Zuul
Branch: stable/train

commit d56765a0aff2f8cf5761a26de86a0ec3a2e5bd0f
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 16 16:41:59 2020 +0100

    Do not force remove containers

    Paunch does docker/podman rm -f, when removes containers. It seems it
    returns too early, having some leftovers (like docker service
    endpoints) behind or pending for it to be removed later, in case of
    big fat deamons.

    Long story short, don't do rm -f and allow it to do its job gracefully
    and without a hurry.

    Change-Id: I346c49cb204f273bd7077ca5153412cda9846534
    Closes-Bug: #1860004
    Co-authored-by: Sergii Golovatiuk <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 3813fc7f2bd5768248f7c450d707a13c33f19fc3)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to paunch (stable/rocky)

Reviewed: https://review.opendev.org/704672
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=de1673979585e971fbb8d38eba1d8308f9bbdd92
Submitter: Zuul
Branch: stable/rocky

commit de1673979585e971fbb8d38eba1d8308f9bbdd92
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 16 16:41:59 2020 +0100

    Do not force remove containers

    Paunch does docker/podman rm -f, when removes containers. It seems it
    returns too early, having some leftovers (like docker service
    endpoints) behind or pending for it to be removed later, in case of
    big fat deamons.

    Long story short, don't do rm -f and allow it to do its job gracefully
    and without a hurry.

    Conflicts:
        paunch/runner.py
        paunch/tests/test_builder_compose1.py
        paunch/tests/test_runner.py

    Change-Id: I346c49cb204f273bd7077ca5153412cda9846534
    Closes-Bug: #1860004
    Co-authored-by: Sergii Golovatiuk <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 3813fc7f2bd5768248f7c450d707a13c33f19fc3)
    (cherry picked from commit d56765a0aff2f8cf5761a26de86a0ec3a2e5bd0f)
    (cherry picked from commit 7a002698561ee65e4e36bf2084043e211ca80ecb)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/paunch 6.0.1

This issue was fixed in the openstack/paunch 6.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to paunch (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/708341

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to paunch (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/708881

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to paunch (stable/stein)

Reviewed: https://review.opendev.org/704669
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=7a002698561ee65e4e36bf2084043e211ca80ecb
Submitter: Zuul
Branch: stable/stein

commit 7a002698561ee65e4e36bf2084043e211ca80ecb
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jan 16 16:41:59 2020 +0100

    Do not force remove containers

    Paunch does docker/podman rm -f, when removes containers. It seems it
    returns too early, having some leftovers (like docker service
    endpoints) behind or pending for it to be removed later, in case of
    big fat deamons.

    Long story short, don't do rm -f and allow it to do its job gracefully
    and without a hurry.

    Change-Id: I346c49cb204f273bd7077ca5153412cda9846534
    Closes-Bug: #1860004
    Co-authored-by: Sergii Golovatiuk <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 3813fc7f2bd5768248f7c450d707a13c33f19fc3)
    (cherry picked from commit d56765a0aff2f8cf5761a26de86a0ec3a2e5bd0f)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to paunch (master)

Reviewed: https://review.opendev.org/708341
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=3006b547cbdc8ae979ffaed0268d5831a576546e
Submitter: Zuul
Branch: master

commit 3006b547cbdc8ae979ffaed0268d5831a576546e
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 18 12:10:57 2020 +0100

    Fallback to a rm -f action for podman

    There is a podman specific behavior that cannot be reproduced with
    docker:

    $ docker run -itd --privileged --name=foo busybox sleep 1000
    $ docker exec -it foo sh -c "sleep 180"
    $ kill -STOP <that process>
    $ docker stop foo
    $ docker rm foo

    works w/o errors, while podman fails the latter step:
    Error: cannot remove container <id> has active exec sessions: container
    state improper

    To make it consistent, add a fallback to re-try it with rm -f.

    Related-bug: #1860004
    Change-Id: Id387f624078ef874aa902656952582c9c54f3f2e
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to paunch (stable/train)

Reviewed: https://review.opendev.org/708881
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=6dfd2d07d32abd078f6cf1590d99af71da98398c
Submitter: Zuul
Branch: stable/train

commit 6dfd2d07d32abd078f6cf1590d99af71da98398c
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Feb 18 12:10:57 2020 +0100

    Fallback to a rm -f action for podman

    There is a podman specific behavior that cannot be reproduced with
    docker:

    $ docker run -itd --privileged --name=foo busybox sleep 1000
    $ docker exec -it foo sh -c "sleep 180"
    $ kill -STOP <that process>
    $ docker stop foo
    $ docker rm foo

    works w/o errors, while podman fails the latter step:
    Error: cannot remove container <id> has active exec sessions: container
    state improper

    To make it consistent, add a fallback to re-try it with rm -f.

    Related-bug: #1860004
    Change-Id: Id387f624078ef874aa902656952582c9c54f3f2e
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 3006b547cbdc8ae979ffaed0268d5831a576546e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/paunch rocky-eol

This issue was fixed in the openstack/paunch rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/paunch queens-eol

This issue was fixed in the openstack/paunch queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/paunch stein-eol

This issue was fixed in the openstack/paunch stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.