filesystem getting full on a contrail analytics node installation
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R4.0 |
Invalid
|
Critical
|
Sundaresan Rajangam | |||
R4.1 |
Invalid
|
High
|
Sundaresan Rajangam | |||
R5.0 |
Invalid
|
High
|
Sundaresan Rajangam | |||
Trunk |
Invalid
|
High
|
Sundaresan Rajangam |
Bug Description
file system is getting full on a contrail analytics node installation.
root@anc1-
Filesystem Size Used Avail Use% Mounted on
udev 16G 4.0K 16G 1% /dev
tmpfs 3.2G 3.4M 3.2G 1% /run
/dev/sda1 280G 251G 15G 95% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 16G 1.4M 16G 1% /run/shm
none 100M 0 100M 0% /run/user
none 280G 251G 15G 95% /var/lib/
shm 64M 0 64M 0% /var/lib/
none 280G 251G 15G 95% /var/lib/
shm 64M 0 64M 0% /var/lib/
none 280G 251G 15G 95% /var/lib/
shm 64M 148K 64M 1% /var/lib/
root@anc1-
241G /var/lib/
4.2G /var/lib/
8.2M /var/lib/
60K /var/lib/
20K /var/lib/
4.0K /var/lib/
4.0K /var/lib/docker/tmp
4.0K /var/lib/
28K /var/lib/
root@anc1-
121G /var/lib/
224K /var/lib/
121G /var/lib/
Inside the controller docker:
root@anc1-
0 /var/lib/
34G /var/lib/
Changed in juniperopenstack: | |
importance: | Undecided → Critical |
tags: | added: analytics |
tags: | added: 2018-0129-0643 jtac |
tags: | added: gci |
Copying comments history from JIRA bug https:/ /aspg-jira. juniper. net/browse/ CXU-18060? filter= 14433
Hi Prasanth,
Customer has 3 CAN nodes running on 3.0 release.
One of the node / partition is 100% hence no containers running on it .
I have logged in to other CAN nodes and ~80G was occupied by /var/crash. That will explain why 80G space was utilized right but on each CAN node has ~280G allocated to / partition. Do you think cleaning up /var/crashes in each container will help the situation ? Also containers not running on node with 100% ( anc1-prd1- csp-adm- 01) / partition , how are we going to clean up space ( assuming you are targeting only /var/crashes) on this anc1-prd1- csp-adm- 01 node.
Please find the attachment for session log.
Thanks
Ram
Prashanth Nageshappa added a comment - 2/15/18 19:58
Clearing core files will help to some extent, but still we need to look at what else is taking ~200GB space.
We will need to have Contrail team debug this to understand why so much space is being taken. Will discuss with Mohan to check who from Contrail team can be involved to look into this.
Prashanth Nageshappa added a comment - 2/15/18 22:19 /bugs.launchpad .net/juniperope nstack/ +bug/1749900
I have opened contrail bug https:/