volume leak within Minikube debian build environment

[Scott Little]

Brief Description
-----------------
Within a Debian/Minikube build environment, volumes are sometimes
orphaned, requiring a 'docker volume prune' the clean up.

CENGN uses a minikube based build environment. It was observed that the
docker volume for minikube had grown to over 200 GB.

docker system df -v
...
Local Volumes space usage:

VOLUME NAME
minikube-jenkins-stx-rc-stx-8-0-debian 1 95.9GB
minikube-jenkins-stx-debian-master 1 74.08GB

Entering the minikube container for minikube-jenkins-stx-rc-stx-8-0-debian, I observed this

docker system df -v
...
Local Volumes space usage:

VOLUME NAME LINKS SIZE
557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb 0 16.48GB
6c71a483b38db6710155a93e867528b4c706cfda7a6725802615fd179b6d9c62 0 22.37GB
7dfe577906908f557d0c03e94d1bc357ad1865179cdbf5c64eea82d0e40b1d38 0 22.35GB
653e260e62cc0c53136870db5b64d81078963e48bf2e26d45cb981d75ac146b7 1 213kB
49ebd2eb6645232668a29d594ac591768d3536545593e468ebd03efd34b889c4 0 23.03GB

So there are four ~20GB volumes that are not being used, suggesting an intermittent issue that orphans volumes (4 orphans from 25 builds).

Now this minikube instance builds nightly, so whatever is orphaning volumes
must be on a failure path.

docker volume inspect 557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb
[
    {
        "CreatedAt": "2023-05-18T03:48:05Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb/_data",
        "Name": "557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb",
        "Options": null,
        "Scope": "local"
    }
]

The 'CreatedAt' date converts to 2023-05-17T23:48:05 in local time

This maps to the run of 'stop-containers' step of the jenkins pipeline that was a 'green' build, with Jenkins logs ....

23:47:56 cd /home/localdisk/designer/jenkins/rc-stx-8-0-debian/repo/stx-tools
23:47:56 sourcing ./import-stx
23:47:59 release "rc-stx-8-0-debian" uninstalled
23:48:11 Deleted Containers:

The pipeline runs this code
...
stx control stop || true

# Prune minikube's docker, then stop minikube's top-level container
if [[ "$STX_PLATFORM" == "minikube" ]] ; then
    profile_args=()
    [[ -z "$MINIKUBENAME" ]] || profile_args+=("-p" "$MINIKUBENAME")
    if minikube "${profile_args[@]}" status >/dev/null ; then
        minikube "${profile_args[@]}" ssh -- 'docker system prune -f'
        minikube "${profile_args[@]}" stop
    fi
fi

Could the "CreatedAt" actually be the last modification time?

Could it be that the "stx control stop" have failed in some way?

At minimum, docker 'system prune -f --volumes' might also be needed in the jenkins pipeline, but that just masks the issue, and only benefits jenkins.

Severity
--------
Minor

Steps to Reproduce
------------------
unclear

Expected Behavior
------------------
Volumes are either deleted or reused. Never orphaned

Actual Behavior
----------------
Volumes are intermittently orphaned

Reproducibility
---------------
Intermittent. 4 out of 25 builds left orphans

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
May 17, 2023

Last Pass
---------
N/A

Timestamp/Logs
--------------
See above

Test Activity
-------------
Build

Workaround
----------
Describe workaround if available

[Scott Little (slittle1)]

Similar leak seen in master branch builds

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/895733

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/895735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/895735
Committed: https://opendev.org/starlingx/jenkins-pipelines/commit/725961efc94e09984d351808a821ff79ece6b4c3
Submitter: "Zuul (22348)"
Branch: master

commit 725961efc94e09984d351808a821ff79ece6b4c3
Author: Davlet Panech <email address hidden>
Date: Mon Sep 18 15:22:53 2023 -0400

    stop-containers: prune minikube volumes

    Prune minikube volumes in addition to orphans & build cache.

    Closes-Bug: 2020807
    Signed-off-by: Davlet Panech <email address hidden>
    Change-Id: If1dcc157e1ce6171779f8143f6fbb7cbd0d02676

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/895733
Committed: https://opendev.org/starlingx/tools/commit/5188f106a066c08ebade01d5dce0f97ad3079929
Submitter: "Zuul (22348)"
Branch: master

commit 5188f106a066c08ebade01d5dce0f97ad3079929
Author: Davlet Panech <email address hidden>
Date: Mon Sep 18 14:57:07 2023 -0400

    stx cleanup: new sub-command

    Add new subcommand to stx tool:

      stx cleanup [--force][--dry-run] [docker|minikube]

    It runs "docker system prune --volumes" in the builder docker instance,
    or in the minikube's docker instance, or both (default).

    Partial-Bug: 2020807
    Signed-off-by: Davlet Panech <email address hidden>
    Change-Id: I35a7a439e6a52455111e5bc3fff1e9377a407085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.8.0
Review: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/895852

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/jenkins-pipelines/+/895852
Committed: https://opendev.org/starlingx/jenkins-pipelines/commit/8c376aa44ea136d8f567293752020ebbbc2439da
Submitter: "Zuul (22348)"
Branch: r/stx.8.0

commit 8c376aa44ea136d8f567293752020ebbbc2439da
Author: Davlet Panech <email address hidden>
Date: Mon Sep 18 15:22:53 2023 -0400

    stop-containers: prune minikube volumes

    Prune minikube volumes in addition to orphans & build cache.

    Closes-Bug: 2020807
    Signed-off-by: Davlet Panech <email address hidden>
    Change-Id: If1dcc157e1ce6171779f8143f6fbb7cbd0d02676
    (cherry picked from commit 725961efc94e09984d351808a821ff79ece6b4c3)