Comment 0 for bug 2020807

Brief Description
-----------------
Within a Debian/Minikube build environment, volumes are sometimes
orphaned, requiring a 'docker volume prune' the clean up.

CENGN uses a minikube based build environment. It was observed that the
docker volume for minikube had grown to over 200 GB.

docker system df -v
...
Local Volumes space usage:

VOLUME NAME
minikube-jenkins-stx-rc-stx-8-0-debian 1 95.9GB
minikube-jenkins-stx-debian-master 1 74.08GB

Entering the minikube container for minikube-jenkins-stx-rc-stx-8-0-debian, I observed this

docker system df -v
...
Local Volumes space usage:

VOLUME NAME LINKS SIZE
557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb 0 16.48GB
6c71a483b38db6710155a93e867528b4c706cfda7a6725802615fd179b6d9c62 0 22.37GB
7dfe577906908f557d0c03e94d1bc357ad1865179cdbf5c64eea82d0e40b1d38 0 22.35GB
653e260e62cc0c53136870db5b64d81078963e48bf2e26d45cb981d75ac146b7 1 213kB
49ebd2eb6645232668a29d594ac591768d3536545593e468ebd03efd34b889c4 0 23.03GB

So there are four ~20GB volumes that are not being used, suggesting an intermittent issue that orphans volumes (4 orphans from 25 builds).

Now this minikube instance builds nightly, so whatever is orphaning volumes
must be on a failure path.

docker volume inspect 557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb
[
    {
        "CreatedAt": "2023-05-18T03:48:05Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb/_data",
        "Name": "557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb",
        "Options": null,
        "Scope": "local"
    }
]

The 'CreatedAt' date converts to 2023-05-17T23:48:05 in local time

This maps to the run of 'stop-containers' step of the jenkins pipeline that was a 'green' build, with Jenkins logs ....

23:47:56 cd /home/localdisk/designer/jenkins/rc-stx-8-0-debian/repo/stx-tools
23:47:56 sourcing ./import-stx
23:47:59 release "rc-stx-8-0-debian" uninstalled
23:48:11 Deleted Containers:

The pipeline runs this code
...
stx control stop || true

# Prune minikube's docker, then stop minikube's top-level container
if [[ "$STX_PLATFORM" == "minikube" ]] ; then
    profile_args=()
    [[ -z "$MINIKUBENAME" ]] || profile_args+=("-p" "$MINIKUBENAME")
    if minikube "${profile_args[@]}" status >/dev/null ; then
        minikube "${profile_args[@]}" ssh -- 'docker system prune -f'
        minikube "${profile_args[@]}" stop
    fi
fi

Could the "CreatedAt" actually be the last modification time?

Could it be that the "stx control stop" have failed in some way?

At minimum, docker 'system prune -f --volumes' might also be needed in the jenkins pipeline, but that just masks the issue, and only benefits jenkins.

Severity
--------
Minor

Steps to Reproduce
------------------
unclear

Expected Behavior
------------------
Volumes are either deleted or reused. Never orphaned

Actual Behavior
----------------
State what is the actual behavior

Reproducibility
---------------
Intermittent. 4 out of 25 builds left orphans

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
May 17, 2023

Last Pass
---------
N/A

Timestamp/Logs
--------------
See above

Test Activity
-------------
Build

Workaround
----------
Describe workaround if available