logs are not logrotated

Bug #1078213 reported by Zygmunt Krynicki on 2012-11-13
100
This bug affects 14 people
Affects Status Importance Assigned to Milestone
juju-core
High
Wayne Witzel III
juju (Ubuntu)
Medium
Unassigned

Bug Description

I've been running a small juju cluster for a week now.

There were some connection issues in the data center I'm using and juju had issues connecting to zookeeper. As a result /var/log/juju/juju-machine-agent.log quickly grew to 200M

Overall on the four machines that make up my installation all of juju logs, combined, were over 2GB in size. With the size of the instances this is quickly becoming an issue. I don't imagine how I would manage a large juju cluster without scripting around this issue. I think that juju must support logrotate automatically out of the box.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: juju 0.5+bzr531-0ubuntu1.3
ProcVersionSignature: User Name 3.2.0-32.51-virtual 3.2.30
Uname: Linux 3.2.0-32-virtual x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 09:09:01 2012
Ec2AMI: ami-000000bf
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: juju
UpgradeStatus: No upgrade log present (probably fresh install)

Zygmunt Krynicki (zyga) wrote :
description: updated
James Page (james-page) wrote :

I've seen big log files in some of my long running deployments as well; Marking 'Triaged' and 'Medium' and raising a task for juju-core as well.

Changed in juju (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
summary: - juju-machine-agent.log is not logrotated
+ juju-machine-agent.log/logs are not logrotated
Changed in juju-core:
milestone: none → 2.0

We clearly should rotate logs, but the urgency has gone down since we made some changes to what we log in the course of initial juju-core scale testing.

Changed in juju-core:
status: New → Triaged
importance: Undecided → Medium
Tim Penhey (thumper) on 2013-06-20
tags: added: logging
Curtis Hovey (sinzui) on 2013-10-11
tags: removed: ec2-images
Curtis Hovey (sinzui) on 2013-10-11
Changed in juju-core:
importance: Medium → High
Curtis Hovey (sinzui) on 2013-10-12
tags: added: canonical-webops canonistack
Tom Haddon (mthaddon) on 2014-03-05
tags: added: canonical-is
removed: canonical-webops
Changed in juju-core:
importance: High → Medium
Curtis Hovey (sinzui) on 2014-06-20
summary: - juju-machine-agent.log/logs are not logrotated
+ logs are not logrotated
Mark Ramm (mark-ramm) on 2014-07-22
Changed in juju-core:
importance: Medium → High
milestone: none → 1.21-alpha1
Ante Karamatić (ivoks) wrote :

$ du -hs /var/log/juju/
6.9G /var/log/juju/

System installed on 2014-06-02.

Ian Booth (wallyworld) on 2014-07-25
Changed in juju-core:
assignee: nobody → Nate Finch (natefinch)
status: Triaged → In Progress
Nate Finch (natefinch) on 2014-08-07
Changed in juju-core:
assignee: Nate Finch (natefinch) → Wayne Witzel III (wwitzel3)
Curtis Hovey (sinzui) on 2014-09-10
Changed in juju-core:
milestone: 1.21-alpha1 → 1.21-alpha2
Haw Loeung (hloeung) wrote :

ubuntu@juju-prod-XX-YYY-machine-0:/var/log/juju$ ls -la
total 19790064
drwxr-xr-x 2 syslog adm 4096 Oct 2 07:33 .
drwxr-xr-x 11 root root 4096 Oct 2 06:38 ..
-rw-r--r-- 1 syslog adm 11719171973 Oct 2 07:33 all-machines.log.1
-rw------- 1 syslog syslog 879 Aug 18 20:17 ca-cert.pem
-rw-r--r-- 1 root root 1757 Oct 2 07:33 machine-0.log
-rw-r--r-- 1 root root 8545817300 Oct 2 07:32 machine-0.log.1
-rw------- 1 syslog syslog 830 Aug 18 20:17 rsyslog-cert.pem
-rw------- 1 syslog syslog 887 Aug 18 20:17 rsyslog-key.pem
u

machines:
  "0":
    agent-state: started
    agent-version: 1.18.4.1
    dns-name: XXX.XXX.XXX.XXX
    instance-id: 542c922e-a61d-4062-8efe-54eed8b9b7b8
    instance-state: ACTIVE
    series: precise
    hardware: arch=amd64 cpu-cores=2 mem=4096M root-disk=51200M

Haw Loeung (hloeung) wrote :

Err, it's .1 because I manually rotated them

As an interim solution:

juju deploy cs:~chris-gondolin/trusty/logrotate
juju set logrotate='logfiles={"juju":{"path":"/var/log/juju/*.log"}}'

On 02/10/14 08:38, Haw Loeung wrote:
> ubuntu@juju-prod-XX-YYY-machine-0:/var/log/juju$ ls -la
> total 19790064
> drwxr-xr-x 2 syslog adm 4096 Oct 2 07:33 .
> drwxr-xr-x 11 root root 4096 Oct 2 06:38 ..
> -rw-r--r-- 1 syslog adm 11719171973 Oct 2 07:33 all-machines.log.1
> -rw------- 1 syslog syslog 879 Aug 18 20:17 ca-cert.pem
> -rw-r--r-- 1 root root 1757 Oct 2 07:33 machine-0.log
> -rw-r--r-- 1 root root 8545817300 Oct 2 07:32 machine-0.log.1
> -rw------- 1 syslog syslog 830 Aug 18 20:17 rsyslog-cert.pem
> -rw------- 1 syslog syslog 887 Aug 18 20:17 rsyslog-key.pem

Wayne Witzel III (wwitzel3) wrote :

Thanks for the Can you provide the contents of the logrotate.conf that Juju generates? There was a bug that prevents Juju from properly rotating log files that was fixed in master (https://github.com/juju/juju/commit/d4328ef1d825679d67aee76f2e6709ce4faddaf6) and I want to make sure that this isn't the same issue.

Curtis Hovey (sinzui) on 2014-10-23
Changed in juju-core:
milestone: 1.21-alpha2 → 1.21-alpha3
Curtis Hovey (sinzui) on 2014-11-04
Changed in juju-core:
milestone: 1.21-alpha3 → 1.21-beta1
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui) on 2014-11-10
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers