logrotate issue: Duplicated path for remote node rsyslogs

Bug #1505180 reported by Ksenia Svechnikova
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Fuel Library (Deprecated)

Bug Description

Version: https://paste.mirantis.net/show/1253/

MOS 8.0 ISO#137

After 2 days of running env we get the /var/log/ filled:

[root@fuel-slab4 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-root 9.6G 995M 8.1G 11% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/vda3 190M 27M 154M 15% /boot
/dev/vda2 200M 256K 200M 1% /boot/efi
/dev/mapper/os-var 12G 12G 0 100% /var
/dev/mapper/os-varlibdocker
                       18G 3.0G 14G 19% /var/lib/docker
/dev/mapper/os-varlog
                      6.6G 6.6G 0 100% /var/log

[root@fuel-slab4 ~]# du -h /var/log/docker-logs/node-/*
90M /var/log/docker-logs/node-/auth.log
3.1M /var/log/docker-logs/node-/cron.log
299M /var/log/docker-logs/node-/daemon.log
188K /var/log/docker-logs/node-/kern.log
232K /var/log/docker-logs/node-/mail.log
1.9G /var/log/docker-logs/node-/messages
2.2G /var/log/docker-logs/node-/syslog
1011M /var/log/docker-logs/node-/user.log

In this directory we have logs from every node, duplicate value of /var/log/docker-logs/remote/node-*.domain.tld:

[root@fuel-slab4 ~]# tail -n 10 /var/log/docker-logs/node-/messages
2015-10-11T22:29:59+00:00 node-2 heat-api-cfn 2015-10-11 22:29:59.338 28045 INFO eventlet.wsgi.server [-] 10.20.14.2 - - [11/Oct/2015 22:29:59] "OPTIONS / HTTP/1.0" 300 274 0.000647
2015-10-11T22:29:59+00:00 node-2 admin 2015-10-11 22:29:59.568 7670 INFO keystone.common.wsgi [-] OPTIONS /?
2015-10-11T22:29:59+00:00 node-1 admin 2015-10-11 22:29:59.603 25315 INFO keystone.common.wsgi [-] OPTIONS /?
2015-10-11T22:29:59+00:00 node-2 cinder-api 2015-10-11 22:29:59.683 6628 INFO eventlet.wsgi.server [-] (6628) accepted ('10.20.14.5', 51537)
2015-10-11T22:29:59+00:00 node-2 cinder-api 2015-10-11 22:29:59.685 6628 INFO cinder.api.openstack.wsgi [-] OPTIONS http://10.20.14.6:8776/
2015-10-11T22:29:59+00:00 node-2 cinder-api 2015-10-11 22:29:59.686 6628 INFO cinder.api.openstack.wsgi [-] http://10.20.14.6:8776/ returned with HTTP 200
2015-10-11T22:29:59+00:00 node-2 cinder-api 2015-10-11 22:29:59.687 6628 INFO eventlet.wsgi.server [-] 10.20.14.5 - - [11/Oct/2015 22:29:59] "OPTIONS / HTTP/1.0" 200 420 0.002744
2015-10-11T22:29:59+00:00 node-1 heat-api-cfn 2015-10-11 22:29:59.840 27725 INFO eventlet.wsgi.server [-] 10.20.14.2 - - [11/Oct/2015 22:29:59] "OPTIONS / HTTP/1.0" 300 274 0.000716
2015-10-11T22:29:59+00:00 node-1 cinder-api 2015-10-11 22:29:59.849 9310 INFO eventlet.wsgi.server [-] (9310) accepted ('10.20.14.5', 51594)
2015-10-11T22[root@fuel-slab4 ~]#

I've attached the snapshot from another lab, because this one can't generate it according to the overfull /var/log/.

Tags: logging
Changed in fuel:
status: New → Confirmed
summary: - Duplicate path for remote node rsyslogs
+ Duplicated path for remote node rsyslogs
Changed in fuel:
importance: Undecided → High
Revision history for this message
Bogdan Dobrelya (bogdando) wrote : Re: Duplicated path for remote node rsyslogs

Please clear some space and provide the logs from the *affected* lab

tags: added: logging
Changed in fuel:
status: Confirmed → Incomplete
summary: - Duplicated path for remote node rsyslogs
+ Docker logs issue: Duplicated path for remote node rsyslogs
summary: - Docker logs issue: Duplicated path for remote node rsyslogs
+ logrotate issue: Duplicated path for remote node rsyslogs
Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

Snapshot couldn't be download to LP. Link to download:
https://drive.google.com/file/d/0B2v38w72jlwTZmZZNzBLVGxqUXM/view?usp=sharing

Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

The issue is renamed wrong. It is not in logrotate, but in rsyslog configuration, so this directory "/var/log/docker-logs/node-/messages" that have all logs from each node.

Logrotation and logs filling is after-effects of this behavior

Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Ksenia, your messages aren't very clear. We've never ever written remote messages to "/var/log/docker-logs/node-/" It goes to /var/log/docker-logs/remote/node-$NODE_ID.$DOMAIN/ and then there is a symlink /var/log/remote -> /var/log/docker-logs/remote. I'm just wondering if logs are appearing in a new directory due to rotation or bad rsyslog configuration.

The google drive link fails at 18mb. Can you possible re-upload it there?

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

I've taken the last snapshot from our CI:

https://product-ci.infra.mirantis.net/job/8.0-kilo.ubuntu.smoke_neutron/125/artifact/logs/pass_deploy_neutron_tun-fuel-snapshot-2015-10-10_00-58-11.tar.xz

and check the path:

fuel-snapshot-2015-10-10_00-58-11$ ls -l 10.109.5.2/var/log/docker-logs/node-/
total 108140
-rw-r--r-- 1 kdemina kdemina 376282 Oct 10 03:58 auth.log
-rw-r--r-- 1 kdemina kdemina 21128 Oct 10 03:58 cron.log
-rw-r--r-- 1 kdemina kdemina 27842501 Oct 10 03:58 daemon.log
-rw-r--r-- 1 kdemina kdemina 44483 Oct 10 03:58 kern.log
-rw-r--r-- 1 kdemina kdemina 2822798 Oct 10 03:58 messages
-rw-r--r-- 1 kdemina kdemina 60593301 Oct 10 03:58 syslog
-rw-r--r-- 1 kdemina kdemina 19015478 Oct 10 03:58 user.log

Revision history for this message
Sergey Novikov (snovikov) wrote :

I can download logs by google drive link. it's works.
Also I confirm this bug. Same behavior meets on my lab.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "138"
  build_id: "138"

Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/233623
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ba2a601daf12c077fb42433313d28e5574db3301
Submitter: Jenkins
Branch: master

commit ba2a601daf12c077fb42433313d28e5574db3301
Author: Alex Schultz <email address hidden>
Date: Mon Oct 12 13:18:25 2015 +0000

    Revert "Moving rsyslog to librarian"

    This reverts commit 949bd5831d8a442940764d358a57f7bc2e67441c.

    Change-Id: I042ee40c2b33dff376101e78145f02199a56dfdf
    Related-Bug: #1505180
    Related-Bug: #1504596

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.