logrotate on master node does not have configuraion for /var/log/docker-logs/remote/...

[Alexey Abashkin]

all logs from discovered nodes are pleaced under /var/log/docker-logs/remote/<NODEsIP>/bootstrap/
these log files aren't mentioned in logrotate configuration:

# grep "/var/log/docker-" /etc/logrotate.d/*
/etc/logrotate.d/10-fuel-docker.conf:"/var/log/docker-*.log"
/etc/logrotate.d/20-fuel-docker.conf:"/var/log/docker-*.log"

After some idle time (10 nodes are dicovered. deployment is not started) we have full /var partition on master node.

[Bogdan Dobrelya (bogdando)]

It is being served by /var/log/remote simlink as well

[Bogdan Dobrelya (bogdando)]

[root@nailgun ~]# less /var/log/remote
lrwxrwxrwx 1 root root 27 Jul 28 09:59 /var/log/remote -> /var/log/docker-logs/remote/
[root@nailgun ~]# find /var/log/remote/ -name "*.log.*"
[root@nailgun ~]# logrotate -f /etc/logrotate.d/10-fuel-docker.conf
[root@nailgun ~]# find /var/log/remote/ -name "*.log.*" | wc -l
355

As you can see forced rotation does rotate remote logs dir

[Alexey Abashkin (aabashkin)]

This is my /etc/logrotate.d/10-fuel-docker.conf (MOS 5.0, fresh install):
==========================================
# managed by puppet

"/var/log/kern.log"
"/var/log/debug"
"/var/log/syslog"
"/var/log/daemon.log"
"/var/log/auth.log"
"/var/log/audit/audit.log"
"/var/log/user.log"
"/var/log/mail.log"
"/var/log/cron.log"
"/var/log/docker"
"/var/log/docker-*.log"
"/var/log/lxc/*.log"
"/var/log/fuelmenu.log"
"/var/log/nailgun-agent.log"
"/var/log/nailgun/*.log"
"/var/log/astute/*.log"
"/var/log/ostf*.log"
"/var/log/httpd/*"
"/var/log/nginx/*.log"
"/var/log/rabbitmq/*"
"/var/log/cobbler/*.log"
"/var/log/supervisor/*.log"
"/var/log/dump.log"
{
# This file is used for daily log rotations, do not use size options here
  sharedscripts
  weekly
  # rotate only if 30M size or bigger
  minsize 30M
  # truncate file, do not delete & recreate
  copytruncate
  # keep logs for 4 rotations
  rotate 4
  # compression will be postponed to the next rotation, if uncommented
  # delaycompress
  compress
  # ignore missing files
  missingok
  # do not rotate empty files
  notifempty
  postrotate
      # issue HUPing for all rsyslog pids including ones in the docker containers, if any.
      /usr/bin/pkill -HUP rsyslogd 2> /dev/null 2> /dev/null || true
      reload rsyslog >/dev/null 2>&1 || true
  endscript
}

==========================================

this is your steps on my master node:

============================================
[root@fuel dockerctl]# find /var/log/remote/ -name "*.log.*"
[root@fuel dockerctl]# logrotate -f /etc/logrotate.d/10-fuel-docker.conf
[root@fuel dockerctl]# find /var/log/remote/ -name "*.log.*" | wc -l
0
============================================

I do not have logs rotated but "disk full".

[Bogdan Dobrelya (bogdando)]

Please update the environment info.

As far as I can see, the issue was fixed both in master and stable/5.0 branches for 5.1 and 5.0.1 releases, but 5.0 release is affected, indeed.

at master branch git blame shows
5319075f (Bogdan Dobrelya 2014-06-16 11:35:05 +0300 25) "/var/log/remote/*-*/*.log"
https://github.com/stackforge/fuel-library/commit/5319075fb440b0880122f64af14056e3808f9d54

at stable/5.0:
94cb3af9 (Bogdan Dobrelya 2014-06-16 11:35:05 +0300 26) "/var/log/remote/*-*/*.log"
https://github.com/stackforge/fuel-library/commit/94cb3af924cffd4433d038280164208a41b48256

Closes bug: #1329991
Closes bug: #1316957

[Bogdan Dobrelya (bogdando)]

looks like the issue is a dup of https://bugs.launchpad.net/fuel/+bug/1329991

[Alexey Abashkin (aabashkin)]

Not exactly.

I think you need to add one more line like

/var/log/remote/*/bootstrap/*log

for "discovered" nodes.

[Bogdan Dobrelya (bogdando)]

You are right, currently bootstrap logs aren't rotated.
If I add "/var/log/remote/*-*/*/*.log"
logrotate -d /etc/logrotate.d/10-fuel-docker.conf 2>&1 | grep bootstrap
shows it is ok now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/110586

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/110599

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110586
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c7bb60af409d5bd51e01d18bdbc96ace5541c43a
Submitter: Jenkins
Branch: master

commit c7bb60af409d5bd51e01d18bdbc96ace5541c43a
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 30 13:04:09 2014 +0300

    Add bootstrap remote logs to rotation

    Current rotation templates are missing the 2nd level
    of directories inside of /var/log/remote/*/
    and that is an issue due to bootstrap logs are residing
    there as well.

    Closes-bug: #1349809

    Change-Id: If8f61220eeb897faf3ba403833564de85718b03d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/110599
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=dec8421a18e3ccfa35978b36b5de3292b53e6b69
Submitter: Jenkins
Branch: stable/5.0

commit dec8421a18e3ccfa35978b36b5de3292b53e6b69
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 30 13:04:09 2014 +0300

    Add bootstrap remote logs to rotation

    Current rotation templates are missing the 2nd level
    of directories inside of /var/log/remote/*/
    and that is an issue due to bootstrap logs are residing
    there as well.

    poke ci2
    Closes-bug: #1349809

    Change-Id: If8f61220eeb897faf3ba403833564de85718b03d
    Signed-off-by: Bogdan Dobrelya <email address hidden>