Excessive disk space taken up by /var/lib/cni/ files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Steven Webster |
Bug Description
Brief Description
-----------------
When a system with multiple pods has been up for a long time (months - years), it can be seen that the amount of files and the space they take in /var/lib/
These files represent a cache of the network attachments to the pods and should be cleaned up when the pod no longer exists. Usually this happens successfully, but in certain circumstances like host reboots, some of the files have been observed to not be cleaned up properly.
Severity
--------
Major: Given enough time, the stale cache files will fill up all available disk space.
Steps to Reproduce
------------------
Create multiple pods, preferably with multiple network attachments per pod. Continually reboot the host and observe the number of files grow in /var/lib/
Expected Behavior
------------------
The files should be cleaned up when the pod id no longer exists.
Actual Behavior
----------------
The number of files can continually grow. If a file has not been selected for automatic garbage collection, it is there forever before manual intervention.
Reproducibility
---------------
It seems to depend on the network attachment definitions and does not occur every time.
System Configuration
-------
N/A
Branch/Pull Time/Commit
-------
master
Last Pass
---------
N/A
Timestamp/Logs
--------------
N/A
Test Activity
-------------
Developer Testing
Workaround
----------
It is possible to cleanup the files manually. Each file has the pod-id (given by crictl ps -v) embedded in it's name. If the file is no longer associated with a running pod-id, it can be safely deleted.
Changed in starlingx: | |
importance: | Undecided → High |
status: | New → Triaged |
assignee: | nobody → Steven Webster (swebster-wr) |
tags: | added: stx.6.0 stx.containers stx.networking |
Fix proposed to branch: master /review. opendev. org/c/starlingx /integ/ +/814439
Review: https:/