/openstack directory getting full

[Satish Patel]

This is most strange issue i have every seen in OSA (CentOS 7.x)

I have OSA running in LAB (playground)

I have OSA running on production.

Today i found my /openstack dir is fulling so rapidly after digging found it has bunch of stuff inside, i thought /openstack is just for logging but something strange going on

[root@ostack-infra-03 /]# du -sh /openstack/*
4.0K /openstack/backup
334M /openstack/log
36K /openstack/ostack-infra-03_ceph-mon_container-a92c1c2a
4.9G /openstack/ostack-infra-03_galera_container-eaacd880
895M /openstack/ostack-infra-03_glance_container-235073d2
0 /openstack/ostack-infra-03_heat_apis_container-17b37df7
0 /openstack/ostack-infra-03_heat_engine_container-d05d1c6e
0 /openstack/ostack-infra-03_horizon_container-74f54bdd
0 /openstack/ostack-infra-03_keystone_container-6a056c78
0 /openstack/ostack-infra-03_memcached_container-11f45bfe
0 /openstack/ostack-infra-03_neutron_agents_container-00edcfd3
0 /openstack/ostack-infra-03_neutron_server_container-daea5dc3
0 /openstack/ostack-infra-03_nova_api_metadata_container-edf3c805
0 /openstack/ostack-infra-03_nova_api_os_compute_container-03bb8ce3
0 /openstack/ostack-infra-03_nova_api_placement_container-1469a861
0 /openstack/ostack-infra-03_nova_conductor_container-e5908a88
0 /openstack/ostack-infra-03_nova_console_container-19da1744
0 /openstack/ostack-infra-03_nova_scheduler_container-69376e50
0 /openstack/ostack-infra-03_rabbit_mq_container-d04ece43
1.9G /openstack/ostack-infra-03_repo_container-77ce64fd
0 /openstack/ostack-infra-03_utility_container-07dd5fae

Lets check why galera using 4.9G

[root@ostack-infra-03 /]# du -sh /openstack/ostack-infra-03_galera_container-eaacd880/*
8.0K /openstack/ostack-infra-03_galera_container-eaacd880/aria_log.00000001
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/aria_log_control
1.1G /openstack/ostack-infra-03_galera_container-eaacd880/galera.cache
2.1M /openstack/ostack-infra-03_galera_container-eaacd880/glance
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/grastate.dat
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/gvwstate.dat
2.3M /openstack/ostack-infra-03_galera_container-eaacd880/heat
784K /openstack/ostack-infra-03_galera_container-eaacd880/horizon
76M /openstack/ostack-infra-03_galera_container-eaacd880/ibdata1
1.0G /openstack/ostack-infra-03_galera_container-eaacd880/ib_logfile0
1.0G /openstack/ostack-infra-03_galera_container-eaacd880/ib_logfile1
16K /openstack/ostack-infra-03_galera_container-eaacd880/innobackup.backup.log
16K /openstack/ostack-infra-03_galera_container-eaacd880/innobackup.move.log
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/innobackup.prepare.log
4.4M /openstack/ostack-infra-03_galera_container-eaacd880/keystone
1.1G /openstack/ostack-infra-03_galera_container-eaacd880/mariadb-bin.000012
692M /openstack/ostack-infra-03_galera_container-eaacd880/mariadb-bin.000013
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/mariadb-bin.index
0 /openstack/ostack-infra-03_galera_container-eaacd880/multi-master.info
11M /openstack/ostack-infra-03_galera_container-eaacd880/mysql
0 /openstack/ostack-infra-03_galera_container-eaacd880/mysql.sock
19M /openstack/ostack-infra-03_galera_container-eaacd880/neutron
14M /openstack/ostack-infra-03_galera_container-eaacd880/nova
4.3M /openstack/ostack-infra-03_galera_container-eaacd880/nova_api
14M /openstack/ostack-infra-03_galera_container-eaacd880/nova_cell0
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/nova_placement
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/ostack-infra-03-galera-container-eaacd880.pid
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/performance_schema
0 /openstack/ostack-infra-03_galera_container-eaacd880/test
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/xtrabackup_galera_info
4.0K /openstack/ostack-infra-03_galera_container-eaacd880/xtrabackup_info

What is going on here?

NOTES: in my LAB /openstack is almost empty and everything working great, what is the role os /openstack dir ?

[Satish Patel (satish-txt)]

Why LXC container is almost empty here

[root@ostack-infra-03 mysql]# ls -l /var/lib/lxc/ostack-infra-03_galera_container-eaacd880/rootfs/var/lib/mysql
total 0

[Satish Patel (satish-txt)]

I think i am royally screwed on production :(

it seem LXC "/dev/mapper/rootvg01-lv04" pointing to /openstack directory :(

How do i fix all this issue, we are live almost.. Damn it...

[root@ostack-infra-03 mysql]# lxc-attach -n ostack-infra-03_galera_container-eaacd880
[root@ostack-infra-03-galera-container-eaacd880 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg01-lv04 432G 28G 405G 7% /
none 492K 0 492K 0% /dev
/dev/mapper/rootvg01-lv01 9.8G 9.5G 299M 98% /var/log
cgroup_root 10M 0 10M 0% /sys/fs/cgroup
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.6G 15G 10% /run
[root@ostack-infra-03-galera-container-eaacd880 /]# ls -l /var/lib/mysql/
total 4980136
-rw-rw---- 1 mysql root 8192 Jun 24 21:31 aria_log.00000001
-rw-rw---- 1 mysql root 52 Jun 24 21:31 aria_log_control
-rw-rw---- 1 mysql root 1073743144 Jul 20 21:19 galera.cache
drwx--S--- 2 mysql root 4096 Jun 24 22:26 glance
-rw-rw---- 1 mysql root 118 Jul 16 23:00 grastate.dat
-rw-rw---- 1 mysql root 264 Jul 16 23:00 gvwstate.dat
drwx--S--- 2 mysql root 4096 Jun 25 00:33 heat
drwx--S--- 2 mysql root 4096 Jun 25 00:42 horizon
-rw-rw---- 1 mysql root 79691776 Jul 20 21:19 ibdata1
-rw-rw---- 1 mysql root 1073741824 Jul 20 21:19 ib_logfile0
-rw-rw---- 1 mysql root 1073741824 Jul 20 21:19 ib_logfile1
-rw-rw---- 1 mysql root 13123 Jun 24 21:35 innobackup.backup.log
-rw-rw---- 1 mysql root 12705 Jun 24 21:31 innobackup.move.log
-rw-rw---- 1 mysql root 3730 Jun 24 21:31 innobackup.prepare.log
drwx--S--- 2 mysql root 4096 Jun 24 22:21 keystone
-rw-rw---- 1 mysql root 1073742094 Jul 14 05:03 mariadb-bin.000012
-rw-rw---- 1 mysql root 724569722 Jul 20 21:19 mariadb-bin.000013
-rw-rw---- 1 mysql root 68 Jul 14 05:03 mariadb-bin.index
-rw-rw---- 1 mysql root 0 Jun 24 21:31 multi-master.info
drwx--S--- 2 mysql root 4096 Jun 24 21:31 mysql
srwxrwxrwx 1 mysql root 0 Jun 24 21:31 mysql.sock
drwx--S--- 2 mysql root 16384 Jun 25 00:04 neutron
drwx--S--- 2 mysql root 12288 Jun 24 22:39 nova
drwx--S--- 2 mysql root 4096 Jun 24 22:38 nova_api
drwx--S--- 2 mysql root 12288 Jun 24 22:39 nova_cell0
drwx--S--- 2 mysql root 20 Jun 24 22:34 nova_placement
-rw-rw---- 1 mysql root 5 Jun 24 21:31 ostack-infra-03-galera-container-eaacd880.pid
drwx--S--- 2 mysql root 20 Jun 24 21:31 performance_schema
drwx--S--- 2 mysql root 20 Jun 24 21:31 test
-rw-rw---- 1 mysql root 47 Jul 16 23:00 xtrabackup_galera_info
-rw-rw---- 1 mysql root 647 Jun 24 21:31 xtrabackup_info
[root@ostack-infra-03-galera-container-eaacd880 /]#

[Jakob Englisch (jakob-englisch)]

>i thought /openstack is just for logging but something strange going on

it's not, it contains all persistant data of your lxc containers, including your mysql database. mysql binlogs are filling up your disk https://mariadb.com/kb/en/library/binary-log/

[Satish Patel (satish-txt)]

Thanks Jakob,

I didn't find that in documentation about how should be your disk layout if you planning for production, mostly we create small root but very large /var disk and in here it created problem for me when /openstack start filling gave me heart attack.

For workaround i move /openstack -> /var/openstack and created symlink to /openstack

Where can i define this variable that dump data on /path/X filesystem instead of /openstack

[Mohammed Naser (mnaser)]

wishlist to add docs to explain that it's not just for logging :)

[Mohammed Naser (mnaser)]

specifically -- https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts.html#configuring-the-storage

[VinceLe (legoll)]

I had the same problem.

I tried to go by the docs with a "lxc" LVM volume for my LXC containers, but it was not working (don't remember why) and got told on IRC that this is not a tested config (LXC over LVM on centos 7) so I should better then stay on something validated.

I disabled LXC over LVM, only to hit this bug, the /openstack filesystem was full.

So +1 for me.