| 2023-05-25 18:05:04 |
Scott Little |
description |
Brief Description
-----------------
Within a Debian/Minikube build environment, volumes are sometimes
orphaned, requiring a 'docker volume prune' the clean up.
CENGN uses a minikube based build environment. It was observed that the
docker volume for minikube had grown to over 200 GB.
docker system df -v
...
Local Volumes space usage:
VOLUME NAME
minikube-jenkins-stx-rc-stx-8-0-debian 1 95.9GB
minikube-jenkins-stx-debian-master 1 74.08GB
Entering the minikube container for minikube-jenkins-stx-rc-stx-8-0-debian, I observed this
docker system df -v
...
Local Volumes space usage:
VOLUME NAME LINKS SIZE
557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb 0 16.48GB
6c71a483b38db6710155a93e867528b4c706cfda7a6725802615fd179b6d9c62 0 22.37GB
7dfe577906908f557d0c03e94d1bc357ad1865179cdbf5c64eea82d0e40b1d38 0 22.35GB
653e260e62cc0c53136870db5b64d81078963e48bf2e26d45cb981d75ac146b7 1 213kB
49ebd2eb6645232668a29d594ac591768d3536545593e468ebd03efd34b889c4 0 23.03GB
So there are four ~20GB volumes that are not being used, suggesting an intermittent issue that orphans volumes (4 orphans from 25 builds).
Now this minikube instance builds nightly, so whatever is orphaning volumes
must be on a failure path.
docker volume inspect 557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb
[
{
"CreatedAt": "2023-05-18T03:48:05Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb/_data",
"Name": "557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb",
"Options": null,
"Scope": "local"
}
]
The 'CreatedAt' date converts to 2023-05-17T23:48:05 in local time
This maps to the run of 'stop-containers' step of the jenkins pipeline that was a 'green' build, with Jenkins logs ....
23:47:56 cd /home/localdisk/designer/jenkins/rc-stx-8-0-debian/repo/stx-tools
23:47:56 sourcing ./import-stx
23:47:59 release "rc-stx-8-0-debian" uninstalled
23:48:11 Deleted Containers:
The pipeline runs this code
...
stx control stop || true
# Prune minikube's docker, then stop minikube's top-level container
if [[ "$STX_PLATFORM" == "minikube" ]] ; then
profile_args=()
[[ -z "$MINIKUBENAME" ]] || profile_args+=("-p" "$MINIKUBENAME")
if minikube "${profile_args[@]}" status >/dev/null ; then
minikube "${profile_args[@]}" ssh -- 'docker system prune -f'
minikube "${profile_args[@]}" stop
fi
fi
Could the "CreatedAt" actually be the last modification time?
Could it be that the "stx control stop" have failed in some way?
At minimum, docker 'system prune -f --volumes' might also be needed in the jenkins pipeline, but that just masks the issue, and only benefits jenkins.
Severity
--------
Minor
Steps to Reproduce
------------------
unclear
Expected Behavior
------------------
Volumes are either deleted or reused. Never orphaned
Actual Behavior
----------------
State what is the actual behavior
Reproducibility
---------------
Intermittent. 4 out of 25 builds left orphans
System Configuration
--------------------
N/A
Branch/Pull Time/Commit
-----------------------
May 17, 2023
Last Pass
---------
N/A
Timestamp/Logs
--------------
See above
Test Activity
-------------
Build
Workaround
----------
Describe workaround if available |
Brief Description
-----------------
Within a Debian/Minikube build environment, volumes are sometimes
orphaned, requiring a 'docker volume prune' the clean up.
CENGN uses a minikube based build environment. It was observed that the
docker volume for minikube had grown to over 200 GB.
docker system df -v
...
Local Volumes space usage:
VOLUME NAME
minikube-jenkins-stx-rc-stx-8-0-debian 1 95.9GB
minikube-jenkins-stx-debian-master 1 74.08GB
Entering the minikube container for minikube-jenkins-stx-rc-stx-8-0-debian, I observed this
docker system df -v
...
Local Volumes space usage:
VOLUME NAME LINKS SIZE
557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb 0 16.48GB
6c71a483b38db6710155a93e867528b4c706cfda7a6725802615fd179b6d9c62 0 22.37GB
7dfe577906908f557d0c03e94d1bc357ad1865179cdbf5c64eea82d0e40b1d38 0 22.35GB
653e260e62cc0c53136870db5b64d81078963e48bf2e26d45cb981d75ac146b7 1 213kB
49ebd2eb6645232668a29d594ac591768d3536545593e468ebd03efd34b889c4 0 23.03GB
So there are four ~20GB volumes that are not being used, suggesting an intermittent issue that orphans volumes (4 orphans from 25 builds).
Now this minikube instance builds nightly, so whatever is orphaning volumes
must be on a failure path.
docker volume inspect 557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb
[
{
"CreatedAt": "2023-05-18T03:48:05Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb/_data",
"Name": "557f6c9fb45724c9f550a23dcba49051588ee2f2af0c0ef930c1bfa40db4e8bb",
"Options": null,
"Scope": "local"
}
]
The 'CreatedAt' date converts to 2023-05-17T23:48:05 in local time
This maps to the run of 'stop-containers' step of the jenkins pipeline that was a 'green' build, with Jenkins logs ....
23:47:56 cd /home/localdisk/designer/jenkins/rc-stx-8-0-debian/repo/stx-tools
23:47:56 sourcing ./import-stx
23:47:59 release "rc-stx-8-0-debian" uninstalled
23:48:11 Deleted Containers:
The pipeline runs this code
...
stx control stop || true
# Prune minikube's docker, then stop minikube's top-level container
if [[ "$STX_PLATFORM" == "minikube" ]] ; then
profile_args=()
[[ -z "$MINIKUBENAME" ]] || profile_args+=("-p" "$MINIKUBENAME")
if minikube "${profile_args[@]}" status >/dev/null ; then
minikube "${profile_args[@]}" ssh -- 'docker system prune -f'
minikube "${profile_args[@]}" stop
fi
fi
Could the "CreatedAt" actually be the last modification time?
Could it be that the "stx control stop" have failed in some way?
At minimum, docker 'system prune -f --volumes' might also be needed in the jenkins pipeline, but that just masks the issue, and only benefits jenkins.
Severity
--------
Minor
Steps to Reproduce
------------------
unclear
Expected Behavior
------------------
Volumes are either deleted or reused. Never orphaned
Actual Behavior
----------------
Volumes are intermittently orphaned
Reproducibility
---------------
Intermittent. 4 out of 25 builds left orphans
System Configuration
--------------------
N/A
Branch/Pull Time/Commit
-----------------------
May 17, 2023
Last Pass
---------
N/A
Timestamp/Logs
--------------
See above
Test Activity
-------------
Build
Workaround
----------
Describe workaround if available |
|
