Docker containers consumes full disk some time after deployment

[bouabid amine]

Hi,

After a successful kolla-ansible deployment, containers consumes rapidly the full disk size

We allocated to the root partition 200GB and a cinder volume of 900G

The installation is based on CENTOS and STEIN version of Openstack

Juste after the deployment, the system "df -h" command indicates that only 11% of the disk is used, but this size grows rapidly until reaching the limit of the disk size, the result of the last command become 100% !

But in the other hand, the command "docker system df" indicates the all the time the following stats! :

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 81 81 9.378GB 2.868GB (30%)
Containers 87 87 629.9MB 0B (0%)
Local Volumes 28 28 205.5GB 0B (0%)
Build Cache 0 0 0B 0B

This installation is based on a all-in-one configuration and attached the globals.yml coniguration file

Is there a mean to control the disk size consumed by containers or may be we need a bigger size for root partition

Best Regards

[Mark Goddard (mgoddard)]

Hello, thanks for your bug report. This is not expected - 200G should be more than enough.

Please provide some further information about where this disk space is being consumed. You can use the 'du' command. For example, you can drill down from the top level of the file system like this:

sudo du -d1 / | sort -h

# Find the largest directory

sudo du -d1 <largest dir> | sort -h

[bouabid amine (ostacker)]

Hello,
Thnaks for your answer
After iterative application of the commande "du -d1" it seems that the /var/lib/docker is using the overall disk space, in particular: volumes and overlay2 directories:

du -d1 -h /var/lib/docker/ | sort -h
0 /var/lib/docker/plugins
0 /var/lib/docker/runtimes
0 /var/lib/docker/swarm
0 /var/lib/docker/tmp
0 /var/lib/docker/trust
16K /var/lib/docker/builder
56K /var/lib/docker/buildkit
284K /var/lib/docker/network
45M /var/lib/docker/image
551M /var/lib/docker/containers
66G /var/lib/docker/overlay2
187G /var/lib/docker/volumes
254G /var/lib/docker/

[bouabid amine (ostacker)]

Hello,
After investigation using the ncdu tool (see at the end of the mail), i identify that the directory /var/lib/docker/kafka is consumming continuously disk space, in a second position the directory kolla_logs

So the solution would be to control the maximum size of these volumes, but what is the best manner to do this?

regards

ncdu 1.14.1 ~ Use the arrow keys to navigate, press ? for help
--- /var/lib/docker/volumes ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   89.5 GiB [##########] /kafka
   34.4 GiB [### ] /kolla_logs
   12.5 GiB [# ] /elasticsearch
  469.2 MiB [ ] /mariadb
  186.1 MiB [ ] /zookeeper
   78.8 MiB [ ] /redis
   49.2 MiB [ ] /influxdb
   12.1 MiB [ ] /glance
    4.8 MiB [ ] /outward_rabbitmq
    4.8 MiB [ ] /rabbitmq
  180.0 KiB [ ] /gnocchi
   84.0 KiB [ ] /monasca_grafana
   44.0 KiB [ ] /openvswitch_db
   44.0 KiB [ ] /nova_compute
   32.0 KiB [ ] metadata.db
   24.0 KiB [ ] /monasca_thresh
   20.0 KiB [ ] /iscsi_info
    8.0 KiB [ ] /nova_libvirt_qemu
    8.0 KiB [ ] /keystone_fernet_tokens
    4.0 KiB [ ] /cinder
    4.0 KiB [ ] /storm
    0.0 B [ ] /libvirtd
    0.0 B [ ] /ceilometer
    0.0 B [ ] /neutron_metadata_socket
    0.0 B [ ] /haproxy_socket
    0.0 B [ ] /sahara
    0.0 B [ ] /nova_libvirt
    0.0 B [ ] /magnum
    0.0 B [ ] /barbican
    0.0 B [ ] /aodh

[bouabid amine (ostacker)]

Hello,
After investigation using the ncdu tool (see at the end of the mail), i
identify that the directory /var/lib/docker/kafka is consumming
continuously disk space, in a second position the directory kolla_logs

So the solution would be to control the maximum size of these volumes,
but what is the best manner to do this?

regards

ncdu 1.14.1 ~ Use the arrow keys to navigate, press ? for help
--- /var/lib/

docker/volumes
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    89.5 GiB [##########] /kafka
    34.4 GiB [### ] /kolla_logs
    12.5 GiB [# ] /elasticsearch
   469.2 MiB [ ] /mariadb
   186.1 MiB [ ] /zookeeper
    78.8 MiB [ ] /redis
    49.2 MiB [ ] /influxdb
    12.1 MiB [ ] /glance
     4.8 MiB [ ] /outward_rabbitmq
     4.8 MiB [ ] /rabbitmq
   180.0 KiB [ ] /gnocchi
    84.0 KiB [ ] /monasca_grafana
    44.0 KiB [ ] /openvswitch_db
    44.0 KiB [ ] /nova_compute
    32.0 KiB [ ] metadata.db
    24.0 KiB [ ] /monasca_thresh
    20.0 KiB [ ] /iscsi_info
     8.0 KiB [ ] /nova_libvirt_qemu
     8.0 KiB [ ] /keystone_fernet_tokens
     4.0 KiB [ ] /cinder
     4.0 KiB [ ] /storm
     0.0 B [ ] /libvirtd
     0.0 B [ ] /ceilometer
     0.0 B [ ] /neutron_metadata_socket
     0.0 B [ ] /haproxy_socket
     0.0 B [ ] /sahara
     0.0 B [ ] /nova_libvirt
     0.0 B [ ] /magnum
     0.0 B [ ] /barbican
     0.0 B [ ] /aodh

Le 13‏/9‏/2019 à 1:32 م, Mark Goddard a écrit :
> Hello, thanks for your bug report. This is not expected - 200G should be
> more than enough.
>
> Please provide some further information about where this disk space is
> being consumed. You can use the 'du' command. For example, you can drill
> down from the top level of the file system like this:
>
> sudo du -d1 / | sort -h
>
> # Find the largest directory
>
> sudo du -d1 <largest dir> | sort -h
>

[Mark Goddard (mgoddard)]

I don't know if docker has a way to limit the size of volumes, or even if we'd want to do that.

I'd suggest trying to work out why your kafka instance is consuming so much space, and which of the log files are growing (I'd guess kafka...)

[bouabid amine (ostacker)]

Hello,

The issue is caused by the MONASCA service, when i disabled it every thing is ok now.

With this service enabled, the kolla_logs volume exceeded 40 G and KAFKA volume grew indefinitly

But what would be the solution if this service is needed ?!

[Mark Goddard (mgoddard)]

I think you need to investigate this yourself. I've given you some suggestions already - check which log files are growing, and what their content is. Is the monasca service actually working correctly? It pushes logs and metrics through kafka, perhaps you have a large volume of these? Do you have debug logging enabled on the system? That will drastically increase the amount of logging.