paunch: reduce container updates downtime

[Emilien Macchi]

Originally reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1795229

When a container needs to be updated (new config_data, new image, etc), it's currently handled in 2 loops:

- a loop that will figure out what containers need to be removed.
- another loop that will create the new containers.

The time between a container deletion and its creation with new parameters can take more than 40 seconds which can be problematic for service downtime.

A way of optimizing it would be to put the process in a single loop.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707408

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/709760

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/711426

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/711432

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707408
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=eb3d3b75bb8bf300fb52390610a6a53ddbfaa48e
Submitter: Zuul
Branch: master

commit eb3d3b75bb8bf300fb52390610a6a53ddbfaa48e
Author: Emilien Macchi <email address hidden>
Date: Wed Feb 12 10:21:38 2020 -0500

    Cleanup containers in the same loop as they are created

    Split delete_missing_and_updated() into 2 methods:

      * delete_missing(), that will remove all containers installed on the
        host but missing from the given config. This runs outside of the
        loop, once.
      * delete_updated(), that will remove a container installed on the host
        (if present), that is part of the config, if config_data changed or
        didn't exist. It runs within the create loop, so the downtime
        between a container removal and creation should be shorter than
        before.
      * make delete_missing(), delete_updated() and rename_containers()
        returning True, if any container has been touched by either. Use
        that flag in order to keep the container_names contents always
        actual.
      * in order to make that cached container_names working and saving off
        extra podman ps/inspect calls, rework it to return a list instead
        of an iterator. There is no huge lists of containers, iterators buy
        us nothing here, while podman CLI calls are the more expensive
        thing and we optimize the latter instead.

    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Change-Id: I3c6d0670e11d035287d12f4207489a13e0891943
    Closes-Bug: #1862954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709760
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=de359d17477996182d1992f689212ea811faebe5
Submitter: Zuul
Branch: stable/train

commit de359d17477996182d1992f689212ea811faebe5
Author: Emilien Macchi <email address hidden>
Date: Wed Feb 12 10:21:38 2020 -0500

    Cleanup containers in the same loop as they are created

    Split delete_missing_and_updated() into 2 methods:

      * delete_missing(), that will remove all containers installed on the
        host but missing from the given config. This runs outside of the
        loop, once.
      * delete_updated(), that will remove a container installed on the host
        (if present), that is part of the config, if config_data changed or
        didn't exist. It runs within the create loop, so the downtime
        between a container removal and creation should be shorter than
        before.
      * make delete_missing(), delete_updated() and rename_containers()
        returning True, if any container has been touched by either. Use
        that flag in order to keep the container_names contents always
        actual.
      * in order to make that cached container_names working and saving off
        extra podman ps/inspect calls, rework it to return a list instead
        of an iterator. There is no huge lists of containers, iterators buy
        us nothing here, while podman CLI calls are the more expensive
        thing and we optimize the latter instead.

    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Change-Id: I3c6d0670e11d035287d12f4207489a13e0891943
    Closes-Bug: #1862954
    (cherry picked from commit eb3d3b75bb8bf300fb52390610a6a53ddbfaa48e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711432
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=1031491ec689f3f965bd189f4d333bcb94ac27fa
Submitter: Zuul
Branch: stable/queens

commit 1031491ec689f3f965bd189f4d333bcb94ac27fa
Author: Emilien Macchi <email address hidden>
Date: Wed Feb 12 10:21:38 2020 -0500

    Cleanup containers in the same loop as they are created

    Split delete_missing_and_updated() into 2 methods:

      * delete_missing(), that will remove all containers installed on the
        host but missing from the given config. This runs outside of the
        loop, once.
      * delete_updated(), that will remove a container installed on the host
        (if present), that is part of the config, if config_data changed or
        didn't exist. It runs within the create loop, so the downtime
        between a container removal and creation should be shorter than
        before.
      * make delete_missing(), delete_updated() and rename_containers()
        returning True, if any container has been touched by either. Use
        that flag in order to keep the container_names contents always
        actual.
      * in order to make that cached container_names working and saving off
        extra podman ps/inspect calls, rework it to return a list instead
        of an iterator. There is no huge lists of containers, iterators buy
        us nothing here, while podman CLI calls are the more expensive
        thing and we optimize the latter instead.

    (cherry picked from commit eb3d3b75bb8bf300fb52390610a6a53ddbfaa48e)
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Change-Id: I3c6d0670e11d035287d12f4207489a13e0891943
    Closes-Bug: #1862954

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711426
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=fab6bd7c18ab447f9b7beca75fb00398534e0359
Submitter: Zuul
Branch: stable/stein

commit fab6bd7c18ab447f9b7beca75fb00398534e0359
Author: Emilien Macchi <email address hidden>
Date: Wed Feb 12 10:21:38 2020 -0500

    Cleanup containers in the same loop as they are created

    Split delete_missing_and_updated() into 2 methods:

      * delete_missing(), that will remove all containers installed on the
        host but missing from the given config. This runs outside of the
        loop, once.
      * delete_updated(), that will remove a container installed on the host
        (if present), that is part of the config, if config_data changed or
        didn't exist. It runs within the create loop, so the downtime
        between a container removal and creation should be shorter than
        before.
      * make delete_missing(), delete_updated() and rename_containers()
        returning True, if any container has been touched by either. Use
        that flag in order to keep the container_names contents always
        actual.
      * in order to make that cached container_names working and saving off
        extra podman ps/inspect calls, rework it to return a list instead
        of an iterator. There is no huge lists of containers, iterators buy
        us nothing here, while podman CLI calls are the more expensive
        thing and we optimize the latter instead.

    (cherry picked from commit eb3d3b75bb8bf300fb52390610a6a53ddbfaa48e)
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Change-Id: I3c6d0670e11d035287d12f4207489a13e0891943
    Closes-Bug: #1862954

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/paunch 6.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/paunch queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/paunch stein-eol release.