size of log files should be controlled

Bug #435137 reported by Jane Silber on 2009-09-23
62
This bug affects 26 people
Affects Status Importance Assigned to Milestone
Ubuntu One Client
Critical
Guillermo Gonzalez
ubuntuone-client (Ubuntu)
High
dobey
Nominated for Karmic by Elliot Murphy

Bug Description

Users are experiencing very large log files (>20GB). AIUI, this is due at least in part to the verbosity level we have set for debugging reasons.

While that level helps us, it's disrespectful of users disk space. Also, the plan to turn down the verbosity level later doesn't really work because that will still allow logs to grow unbounded over time.

I think we need to come up with a better solution - i.e,. bound the max size of a log file, overwrite it each day, etc?

Related branches

Joshua Blount (jblount) wrote :

I saw this recently on the twitternet: http://twitter.com/muffinresearch/status/4142427805 which prompted this discussion.

affects: ubuntuone-servers → ubuntuone-client
tags: added: foundations+
Changed in ubuntuone-client:
assignee: nobody → Lucio Torre (lucio.torre)
importance: Undecided → High
status: New → Triaged

bzr runs two log files. ~/.bzr.log, and .bzr.log.old. We cap them at a
certain size, and rotate log ->old when we exceed that size.

This works pretty well.

-Rob

Joshua Blount (jblount) on 2009-09-24
visibility: private → public
Aldiantoro Nugroho (kriwil) wrote :

maybe bound the max size and compress it?

Lucio Torre (lucio.torre) wrote :

We need to:
- lower the log level for syncdaemon
- make sure we compress old log files
- make sure we delete older log files

configuring logrotate and syncdaemon with the correct default values should be enough.

Changed in ubuntuone-client:
assignee: Lucio Torre (lucio.torre) → Guillermo Gonzalez (verterok)
Guillermo Gonzalez (verterok) wrote :

Ok, let's talk a bit about this :)

> We need to:
> - lower the log level for syncdaemon
So should we move the log level to INFO?
btw, this implies that we aren't going to be able to debug what's going on.

> - make sure we compress old log files
This can be done, I already wrote code that do this using python logging framework for the server side, so doing this in the client is easy

> - make sure we delete older log files
We are currently rotating old log files, on a daily basis, if this isn't working there is a bug in the syncdaemon logger module.
We can change this to rotate the log files based on it's size (also supported by the logging framework)

>
> configuring logrotate and syncdaemon with the correct default values should be enough.
logrotate isn't an option, as the syncdaemon is a per user process, and logrotate only have a system wide config. so we will need one logrotate config for eahc user...

Guillermo Gonzalez (verterok) wrote :

Oh, forgot to mention that we are rotating the logs on each startup, and we keep the last 5 log files, logs older than that are discarded.

Steve Alexander (stevea) wrote :

If we're rotating logs only on start-up, that's a problem. It could be days and lots of operations before restarting.

Facundo Batista (facundo) wrote :

Steve: we are currently rotating old log files, on a daily basis, *and* on start-up.

Anthony Hook (anthonyhook) wrote :

My girlfriend had a notification that her /home directory was full. After checking Disk Usage Analyzer, I found her .cache/ubuntuone/log/syncdaemon.log.2009-10-04_14-28-55 file was 39.8gb's. This can cause serious file system problems on something that doesn't have a lot of hard disc space (for instance a netbook) and this is a very severe problem.

Anthony Hook (anthonyhook) wrote :
Anthony Hook (anthonyhook) wrote :
dobey (dobey) on 2009-10-06
tags: added: ubuntuone-karmic
Elliot Murphy (statik) wrote :

The log level needs to be turned way down in the next upload, and if we aren't already rotating logs they need to be rotated.

tags: added: critical
Guillermo Gonzalez (verterok) wrote :

Elliot,
Just for the record.
The requested fix is to change the log level to ERROR

Elliot Murphy (statik) on 2009-10-06
tags: removed: critical
Changed in ubuntuone-client:
importance: High → Critical
Changed in ubuntuone-client (Ubuntu):
assignee: nobody → Guillermo Gonzalez (verterok)
Changed in ubuntuone-client:
milestone: none → w25-karmic-finalfreeze
Changed in ubuntuone-client:
status: Triaged → In Progress
Guillermo Gonzalez (verterok) wrote :

branch proposed and approved, that changes the log level to info and limit the log file size to 1MB.
Now the log file is rotated:
 - when it size is 1MB
 - daily
 - on each daemon start

And the backup cound is 5

Changed in ubuntuone-client:
status: In Progress → Fix Committed
dobey (dobey) on 2009-10-08
Changed in ubuntuone-client (Ubuntu):
milestone: none → ubuntu-9.10
assignee: Guillermo Gonzalez (verterok) → Rodney Dawes (dobey)
importance: Undecided → High
status: New → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntuone-client - 1.0.0-0ubuntu1

---------------
ubuntuone-client (1.0.0-0ubuntu1) karmic; urgency=low

  * New upstream release.
    - Limit log messages, file size, and rotate more often (LP: #435137)
    - Rotate oauth-login.log and cap the file size (LP: #445514)
    - Remove pycurl usage in favor of urllib and better errors (LP: #411029)
    - Save the syncdaemon settings for bw limiting (LP: #418882)
    - Fix NoSuchDatabase when pairing desktopcouch (LP: #438411)
    - Fix IOError when launching preferences window (LP: #441039)
    - Update icons to remove Ubuntu logo, better names (LP: #434886)
    - Use ngettext for the possibly singular notification (LP: #449269)
    - Only create Places bookmark when first authorizing (LP: #397749)
    - Don't always recreate the bookmark (LP: #401211)
    - Handle showing emblems better (LP: #440839)
    - Remove the System->Preferences menu item (LP: #443342)
    - Remove NotOnlineError usage (LP: #404550)
    - Set a default share name in the Nautilus sharing UI (LP: #369488)
    - Simplify some state handling (LP: #420354)

 -- Rodney Dawes <email address hidden> Mon, 28 Sep 2009 18:15:00 -0400

Changed in ubuntuone-client (Ubuntu):
status: In Progress → Fix Released
dobey (dobey) on 2009-10-13
Changed in ubuntuone-client:
status: Fix Committed → Fix Released
luchon (ichekryg) wrote :

I am having similar issue with ubuntuone-client, specifically with ubuntuone-syncd cpu usage spikes.
I have a single 2.7K file on ubuntuone. CPU usage jumps between 8%-40% continuously.
...
ubuntuone-client-1.0.3-0ubuntu1
...
python ubuntuone-client-diagnose.py
Checking your Ubuntu One client...
No issues were detected.
...
 ls -l .cache/ubuntuone/log/

-rw-r--r-- 1 user user 559362 2010-03-09 21:09 oauth-login.log
-rw-r--r-- 1 user user 1048445 2010-03-09 16:25 oauth-login.log.1
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-14
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-15
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-17
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-18
-rw-r--r-- 1 user user 1067 2010-03-09 23:04 syncdaemon-exceptions.log.2010-03-09_23-04-19
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-14
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-15
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-17
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-18
-rw-r--r-- 1 user user 2606 2010-03-09 23:04 syncdaemon.log.2010-03-09_23-04-19
...
cat .cache/ubuntuone/log/syncdaemon-exceptions.log
2010-03-09 23:05:22,607 - ubuntuone.SyncDaemon.Main - ERROR - Local rescan finished with error: Traceback: <type 'exceptions.KeyError'>: ('', 'None')
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/local_rescan.py:139:_process_next_queue
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:243:callback
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:312:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.6/dist-packages/twisted/internet/defer.py:328:_runCallbacks
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/local_rescan.py:77:_process_trash
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/filesystem_manager.py:1055:get_iter_trash
/usr/lib/python2.6/UserDict.py:110:iteritems
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:201:__getitem__
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:132:__getitem__
/usr/lib/python2.6/dist-packages/ubuntuone/syncdaemon/file_shelf.py:124:_load_pickle

PeterPall (peterpall) wrote :

I was always wondering what spins up my hard disk every 2 minutes. Now I created a file named ~/.config/ubuntuone/logging.conf with the following contents:

[logging]
level = ERROR

Did have to restart ubuntuone sync to make the change being applied (u1sdtool -q; u1sdtool -c). But afterwards the problem was fixed for me.

dobey (dobey) wrote :

PeterPall, I suspect you are hitting a completely different issue. Probably bug #765441.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers