influxdb writing data files to rootfs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Eric MacDonald |
Bug Description
Brief Description
-----------------
Observed continuous growth in rootfs usage on active controller over a week when running in steady state. Total growth was 840MB, source of growth was /var/lib/
Severity
--------
Major
Steps to Reproduce
------------------
- install system
- periodically monitor rootfs disk usage via "df -m" and "du -x --max-depth=3 /"
Expected Behavior
------------------
rootfs usage should remain stable.
Actual Behavior
----------------
rootfs usage is steadily increasing over time
Reproducibility
---------------
Reproducible
System Configuration
-------
Multi-node (2+10)
Branch/Pull Time/Commit
-------
cengn load: 20190421T233001Z
Last Pass
---------
New test
Timestamp/Logs
--------------
Relevant files found in /var/lib/
controller-
Thu May 2 00:44:00 UTC 2019
/var/lib/
total 4
drwx------ 2 influxdb influxdb 4096 May 2 00:00 monitor
/var/lib/
total 246492
-rw-r--r-- 1 influxdb influxdb 16777216 Apr 24 23:59 4
-rw-r--r-- 1 influxdb influxdb 33554432 Apr 26 00:00 6
-rw-r--r-- 1 influxdb influxdb 33554432 Apr 26 23:59 8
-rw-r--r-- 1 influxdb influxdb 33554432 Apr 27 23:59 10
-rw-r--r-- 1 influxdb influxdb 67108864 Apr 29 00:00 12
-rw-r--r-- 1 influxdb influxdb 67108864 Apr 29 23:59 14
-rw-r--r-- 1 influxdb influxdb 67108864 Apr 30 23:59 16
-rw-r--r-- 1 influxdb influxdb 67108864 May 2 00:00 18
-rw-r--r-- 1 influxdb influxdb 4194304 May 2 00:44 20
/var/lib/
total 4
drwx------ 2 influxdb influxdb 4096 May 2 00:00 collectd samples
/var/lib/
total 620140
-rw-r--r-- 1 influxdb influxdb 33554432 Apr 24 00:00 2
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 25 00:00 3
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 26 00:00 5
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 27 00:00 7
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 28 00:01 9
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 29 00:02 11
-rw-r--r-- 1 influxdb influxdb 134217728 Apr 30 00:00 13
-rw-r--r-- 1 influxdb influxdb 134217728 May 1 00:00 15
-rw-r--r-- 1 influxdb influxdb 134217728 May 2 00:00 17
-rw-r--r-- 1 influxdb influxdb 8388608 May 2 00:43 19
actual disk usage per file:
controller-
Thu May 2 00:44:09 UTC 2019
32 /var/lib/
72 /var/lib/
72 /var/lib/
71 /var/lib/
72 /var/lib/
72 /var/lib/
71 /var/lib/
69 /var/lib/
72 /var/lib/
8 /var/lib/
606 /var/lib/
606 /var/lib/
14 /var/lib/
19 /var/lib/
23 /var/lib/
28 /var/lib/
33 /var/lib/
37 /var/lib/
42 /var/lib/
47 /var/lib/
3 /var/lib/
241 /var/lib/
241 /var/lib/
847 /var/lib/
controller-1 went active around 20:00 on Apr. 23 and remained active.
Logs available on request.
Test Activity
-------------
System Engineering
Marking as release gating as the rootfs gets filled up over time. To mitigate this issue, the retention period for influxdb will be reduced.