logrotate not rotates logs for named nodes

[Alexander Evseev]

# fuel --fuel-version
api: '1.0'
astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
auth_required: true
build_id: 2015-03-26_21-32-43
build_number: '233'
feature_groups:
- mirantis
fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
production: docker
python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 3f1ece0318e5e93eaf48802fefabf512ca1dce40
      build_id: 2015-03-26_21-32-43
      build_number: '233'
      feature_groups:
      - mirantis
      fuellib_sha: 9c7716bc2ce6075065d7d9dcf96f4c94662c0b56
      fuelmain_sha: 320b5f46fc1b2798f9e86ed7df51d3bda1686c10
      nailgun_sha: b163f6fc77d6639aaffd9dd992e1ad96951c3bbf
      ostf_sha: a4cf5f218c6aea98105b10c97a4aed8115c15867
      production: docker
      python-fuelclient_sha: e5e8389d8d481561a4d7107a99daae07c6ec5177
      release: '6.1'

Logrotate contains rules only for nodes without any names, i.e. beginning with number (IP address?):

[root@fuel ~]# grep -r remote /etc/logrotate.*
/etc/logrotate.d/10-fuel-docker.conf:"/var/log/remote/[1-9]*/*.log"
/etc/logrotate.d/10-fuel-docker.conf:"/var/log/remote/[1-9]*/*/*.log"
/etc/logrotate.d/20-fuel-docker.conf:"/var/log/remote/[1-9]*/*.log"
/etc/logrotate.d/20-fuel-docker.conf:"/var/log/remote/[1-9]*/*/*.log"

And logs for named nodes have very huge sizes:

1837571 527888 -rw-r--r-- 1 root adm 540549120 Апр 3 07:17 /var/log/docker-logs/remote/node-37.domain.tld/neutron-server.log

More than 500MBytes!

[Alexander Evseev (aevseev)]

One more example from environment with small activity:

[root@fuel ~]# find /var/log -size +100M -ls
1049255 105740 -rw-r--r-- 1 root adm 108269391 Апр 3 07:38 /var/log/docker-logs/remote/node-9.domain.tld/keystone-all.log
1049103 107020 -rw-r--r-- 1 root adm 109573450 Апр 3 07:38 /var/log/docker-logs/remote/node-7.domain.tld/keystone-all.log

[Dina Belova (dbelova)]

Critical for the scale lab, we do not have endless disks, they're just finished after two runs... We do not have so much time to redeploy clusters after every rally run.

[Bartłomiej Piotrowski (bpiotrowski)]

Can you show us the output of these two commands?
1. ls -lh /var/log/remote
2. /usr/sbin/logrotate -v /etc/logrotate.d/20-fuel*.conf

[Bogdan Dobrelya (bogdando)]

This issue is likely a dupplicate of https://bugs.launchpad.net/fuel/+bug/1428150
Please provide the output from the fuel master node:
ls -ld /var/log/remote/* | awk '{ print $11 }' | sort | uniq -d

[Alexander Evseev (aevseev)]

No. /var/log/remote/ has not only symlinks:

[root@fuel ~]# ll /var/log/remote/
…
lrwxrwxrwx 1 root root 19 Апр 3 10:20 10.20.1.199 -> node-199.domain.tld
lrwxrwxrwx 1 root root 17 Апр 3 10:20 10.20.1.2 -> node-1.domain.tld
lrwxrwxrwx 1 root root 18 Апр 3 10:20 10.20.1.20 -> node-20.domain.tld
lrwxrwxrwx 1 root root 19 Апр 3 10:20 10.20.1.200 -> node-200.domain.tld
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.201
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.202
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.203
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.204
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.205
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.206
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.207
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.208
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.209
lrwxrwxrwx 1 root root 18 Апр 3 10:20 10.20.1.21 -> node-21.domain.tld
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.210
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.211
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.212
drwxr-xr-x 3 root root 4096 Апр 3 09:21 10.20.1.213
…

[Alexander Evseev (aevseev)]

One more example (size chosen to display not much results):

[root@fuel ~]# find /var/log/remote/ -size +93M -ls
263996 95528 -rw-r--r-- 1 root adm 97779930 Апр 3 12:07 /var/log/remote/node-138.domain.tld/neutron-openvswitch-agent.log
132310 102340 -rw-r--r-- 1 root adm 104766212 Апр 2 07:42 /var/log/remote/node-92.domain.tld/neutron-l3-agent.log
262734 97272 -rw-r--r-- 1 root adm 99568120 Апр 3 12:07 /var/log/remote/node-111.domain.tld/neutron-openvswitch-agent.log

Note that this list doesn't contains symlinks to node-138…

[Alexander Evseev (aevseev)]

[root@fuel ~]# ls -lh /var/log/remote
lrwxrwxrwx 1 root root 27 Апр 1 08:51 /var/log/remote -> /var/log/docker-logs/remote

Output of "logrotate -v" is huge and attached as file.

[Alexander Evseev (aevseev)]

"find" by default don't follows symlinks and therefore I don't seen duplicates.

It seems that because of rally tests cluster generates a lot of logs in periods between logrotate runs (1 hour).
Not Fuel issue.
Sorry.

[Tomasz 'Zen' Napierala (tzn)]

Lowered to High, as it is not critical according to Fuel bug classification rules