[lucid] Ubuntuone-syncdaemon using enormous amounts of ram

[Jeff Lane ]

Binary package hint: ubuntuone-client

I was looking at system resource utilization and noticed that my system RAM is nearly completely used... I checked via system monitor and noticed that the Ubuntuone-Syncdaemon process is using 731MB... It's been running for a while, and currently the state via u1dstool is:

bladernr@klaatu:~$ u1sdtool -s
State: SERVER_RESCAN
    connection: With User With Network
    description: doing server rescan
    is_connected: True
    is_error: False
    is_online: False
    queues: WORKING_ON_BOTH

The only thing that exists in my U1 Storage is a few text files and a few mp3s I purchased via the Music Store... I can't think of any reason why the syncdaemon should be using up nearly 1GB of RAM, especially while it seems to just be sitting there.

Checking current-transfers shows:
bladernr@klaatu:~/Ubuntu One$ u1sdtool --current-transfers
Current uploads: 0
Current downloads: 0

Here's it's listing via ps auxf:
bladernr 2523 1.4 19.7 974496 799148 ? SLl Apr20 31:44 /usr/bin/python /usr/lib/ubuntuone-client/ubuntuone-syncdaemon

And in the time it's taken for me to write this bug report, memory usage has gone from 731MB to 758MB

There appears to be a memory leak in the syncdaemon.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: ubuntuone-client 1.2.0-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.32-21.32-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic x86_64
NonfreeKernelModules: nvidia
Architecture: amd64
CheckboxSubmission: 89874cc6062c150ee1cec9632b63a0a3
CheckboxSystem: 5f30ac82cc48ed91bb5240b61cb4e295
Date: Thu Apr 22 09:57:16 2010
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: ubuntuone-client
UbuntuOneClientConfig:
 [ubuntuone]
 bookmarked = True
 connected = False
 connect = 0
 show_applet = 0
UbuntuOneUserSyncdaemonConfig:
 [bandwidth_throttling]
 read_limit = 2097152
 write_limit = 0
 on = False

[Jeff Lane  (bladernr)]

Here's a tarball with info from /proc/PID for the massive syncdaemon instance.

[Jeff Lane  (bladernr)]

And I thought I'd attach a coredump taken with gcore in case it helps...

[Facundo Batista (facundo)]

Hi!

We would need DEBUG logs to find the issue here.

You should do the following:

1. stop the syncdaemon client and be sure it's fully stopped ("ps -eaf | grep ubuntuone-client" should give you nothing).

2. put a file named syncdaemon.conf in your $HOME/.config/ubuntuone directory with the following information:

[__main__]
log_level = DEBUG

3. restart the client.

4. attach here the logs, just zip you $HOME/.cache/ubuntuone/log/ folder and attach the zip here.

Thanks for your time and help!

[Jeff Lane  (bladernr)]

Facundo: After killing syncdaemon to set the log level, I've been unable to recreate whatever it was that caused it to originally chew through all that ram. I've been running it now for a while, daily, and have synced nearly 8GB of data between my system and U1 without issue at this point.

So whatever it was that caused this was either fixed in an update, or maybe was just a transient issue that is just going to be impossible to reliably re-create...

So I'll let you decide what to mark this (invalid, or whatever) ...

Cheers

Jeff

[Facundo Batista (facundo)]

Jeff, thanks for the info.

Note that when ubuntone-client is updated it is not restarted, so maybe you got one of the so many fixes we did for this, and only when you restarted you actually started to using them...

In any case, I'm marking this bug as invalid, as we can not reproduce it again and we don't have enough information to proceed.

Feel free to reopen it or open a new bug if something like this happens to you again.

Remember that if you have any doubt it may be quicker to contact us in the #ubuntuone IRC channel, on Freenode. You can ping me directly, my nick is facundobatista.

Thanks for your time and help!

[Scott Ritchie (scottritchie)]

Twice in the past few days I've returned from my computer at screensaver barely able to log back in. The disk was thrashing and ubuntuone-syncdaemon was using up several gigabytes of memory (more than my entire shared folder).

The memory leak, intermittent as it is, is definitely alive in Lucid final. I'll try the above debug suggestion and see if I can catch it again.

[Duane Hinnen (duanedesign)]

I noticed in the original comments the syncdaemon.conf had a write value of 0

UbuntuOneUserSyncdaemonConfig:
 [bandwidth_throttling]
 read_limit = 2097152
 write_limit = 0
 on = False

Could this have caused the issue? Did you replace the syncdaemon.conf as per comment #4. What is the current Read/ Write values set in ~/.config/ubuntuone/syncdaemon.conf

thank you,
duanedesign

[Facundo Batista (facundo)]

Thanks Scott, I'll wait for the debug logs.

[Scott Ritchie (scottritchie)]

And it happened again today, attached is my entire .cache/ubuntuone log folder.

The problem seems to happen about once a week, on average, making my system completely unusable. For most users this will cause occasional data loss if anything was left unsaved before leaving their computer unattended for a bit.

[Scott Ritchie (scottritchie)]

duanedesign the writelimit is not the issue, here is my syncdaemon.conf:

[bandwidth_throttling]
read_limit = 2097152
write_limit = 2097152
on = False

[Scott Ritchie (scottritchie)]

Happened again today. Do you want another log?

[Jeff Lane  (bladernr)]

Here is a tarball of the logs dir run after adding debug in the U1 config...

When I killed the syncdaemon this time, it was using up about 2.8GB out of 4, so roughly 75% of my total RAM was being chewed up by U1SD and it was just thrashing my hard disk swapping things in and out...

[Jeff Lane  (bladernr)]

duanedesign requested and I provide :-)

Here is a tarball of the .cache/ubuntuone/log directory from my attempt last night. For context, it was suggested that I change writelimit to 2048 in the config file. So I did so, and still found that after a while, u1sd was using up roughly 75% of system RAM (about 3GB in total). This seems to happen every time I run ubuntuone-syncd now and I am wondering if the U1 servers are timing out or just not happy that I have so much stuff stored... there seems to always be connection error messages in the debug logs that I notice after the syncdaemon starts chewing up RAM.

[Jeff Lane  (bladernr)]

changed to high because A: U1 is now all but unusable due to this issue occurring every time I run ubuntuone-syncdaemon and B: it makes the entire system all but unusable. If I catch it early enough, I can kill off the syncdaemon process and recover my system, however, if this happens overnight, by the time I wake in the morning, the system is so bogged down due to the memory leakage that I have to power cycle to get the system into a usable state.

[Jeff Lane  (bladernr)]

This happens EVERY time syncdaemon runs now. It's getting hung on something, perhaps, but either way, it's leaking like a sieve and chewing up enormous amounts of ram... I've let it get as high as 3GB out of 4GB total. Now I just immediately kill UbuntuOne as soon as I boot the system.

Is there ANY traction at all on this? The only step I have left at this point is to just give up on U1 and purge all the packages... It's a nice idea, and seemed pretty cool, but at this point, it's utterly unusable and actually degrades system performance to the point where the system itself is unusable.

[Roman Yepishev (rye)]

Facundo, from Jeff's debug log I see that the content queue is 36114, metadata queue is 8473. While content queue is large, it was not a memory issue earlier. Is it possible that hash queue does not free the memory that was used to hash file contents?

[Facundo Batista (facundo)]

Roman: The act of hashing barely uses any memory, as the file is not read completely to hash it, it's done by chunks.

I tried to reproduce this issue, and I saw a memory increment when making the queues bigger. However, not that much as in this case (I created a content of 7000 and a meta queue of 7500, and the memory increment was of 10MB).

I saw in this bug and other one that the worst memory usage cases were while SERVER_RESCAN. The process of SERVER_RESCAN is a special case of this, because in old clients it involved a massive queue filling (because each change generated listdirs or downloads, and each listdir was a directory to download, etc.).

The new client that is in Maverick, supports a feature called "generations". Not only the SERVER_RESCAN step is much much simpler (and a lot faster), but it handles simpler queues.

Also, I saw that some memory peaks are found while LocalRescan finds a zillion changes. I just found that this could be improved, because LR sent to hash not only when the file was modified, but also when it was accessed (there is a branch proposed for this).

Furthermore, I'll work on reducing the memory footprint of the content and meta queues (which will have an impact not only while scanning, but on the rest of the client work).

[Santiago Gala (sgala)]

I have the intuition that it happens when the network connection is on/off a number of times, either because the laptop "roams" from one wifi to another, because it sleeps/wakes up several times, or because the server it is connected to is restarted,...

[Jeff Lane  (bladernr)]

Santiago: In my case, your intuition is incorrect. This happens regardless of circumstance for me. FWIW, the standard setup for this particular machine, when I am not traveling, is connected via copper to my LAN with wi-fi turned off, on AC power with no suspend or hibernation. My AC power settings disable all that so when on AC the system doesn't even sleep when the lid is closed. It stays on AC pretty much 24/7 at home, with an active internet connection.

It COULD be, perhaps, that during the server scan, something is causing packet loss or some other issue in transit between my home and the cloud, causing the local client to never actually complete what it's doing, or something like that... but at the very least, I can say that it's not roaming between APs and does not suspend or hibernate.

I'm thinking that maybe the network connection times out and that causes the client to go a bit nuts, as mentioned above, but that's just a general hypothesis based on the server timeout messages that always seem to appear when the memory leakage is at its worse (at least that's usually when I start looking in the logs and notice them.)

Also, Facundo, Roman: FWIW, I have the 50GB U1 account and do have a good bit stored there (not 50GB, but certainly more than 2). Could this be related to having that much data stored in the cloud? The U1 dashboard online says I'm using 7.6GB more or less...

[Facundo Batista (facundo)]

Jeff, yes, it could be very related. Right now, the process of SERVER_RESCAN is O(N) in function of your data. So, if you have a lot of data, SERVER_RESCAN will take a lot, and the three details I found about memory usage also will be impacted about that.

The good news is that SERVER_RESCAN right now in the new 'generations' client is O(1), so no more impact here, and I'm working on fixing those details I found (two are small memory usage that could be less, and one is a leak that always added info to a container and never cleaned it; the later is hard to fix, but I'll go for it).

[Scott Ritchie (scottritchie)]

I'll note I'm also getting occasional .u1conflict files (even though I'm using only a single machine), likely because the daemon has to get killed every now and again and gets confused.

[Facundo Batista (facundo)]

Checked it with heappy, and now it does not leak what I found any more.

These changes are about revno 667, that I think should get into beta anytime soon (they will be in Maverick, of course).

[Jeff Lane  (bladernr)]

Facundo,

I've been running with Maverick Beta so far and memory usage has stayed at about 520MB, give or take a few MB according to top.

The leaking has certainly been resolved.

This is as of whatever bits were in Beta and slightly beyond Beta.

One thing I've noticed, however, is that now I have to manually do a u1sdtool -c to get it to actually connect (it's not doing that anymore at boot time)... but I'm not sure if that's because of something I've mucked with while trying to sort this mess out, or if it's a new issue with the U1 client...

[Facundo Batista (facundo)]

Jeff, it's great to hear that this is working ok in real life for you too! Thanks for reporting back!

Regarding the not automatic connection at startup, I really don't know if that is desired or a bug... Beyond that, did you heard about Magicicada?

    https://edge.launchpad.net/magicicada

It will allow you to have control about the client, seeing its state and everything...

[Radu Marginean (radu-marg)]

Using Ubuntu 11.04. When U1 sync kicks in it consumes up all available ram until my laptop becomes unresponsive. This bug renders Ubuntu One useless ..

[Reinis Zumbergs (reinis-zumbergs)]

I have a situations very much the same in Natty. I am on Lubuntu 64 bit.
First time installed ubuntuone. Found out that I have no way to add folders from Ubuntu One Control Panel(!?). Was forced to install Nautilus. Marked some .folders to backup configuration (one after previous is synced, because sooner it wouldn't respond to marking in nautilus context menu). Went to sleep.

In the morning my computer was almost completely irresponsive, because all the free ram was eaten by syncdaemon:
roodis@rootnis:~$ ps auxw|grep ubuntuone
roodis 4737 0.0 0.0 88572 1060 pts/1 S+ 10:10 0:00 grep --color=auto ubuntuone
roodis 13738 0.6 32.1 1071552 660168 ? Sl Jul03 4:40 /usr/bin/python /usr/lib/ubuntuone-client/ubuntuone-syncdaemon
As it reads - over 600 MiB - 32% of RAM

Closed some programs, started to look around how to report this bug. An hour later, now, that it has more free RAM, it has reached 740 MiB.

[Jeff Lane  (bladernr)]

I hate to say this, but this is still broken in Natty, just as Reinis and Radu state.

On my system (the same system that I opened this bug initially with Lucid), which is running Natty with all latest updates, if I let Ubuntu One run, eventually, the syncdaemon consumes 99% of at least one core on my quad core processor, and I've seen memory usage as high as 3.4 GB out of 4.

I'm at the point now where I've removed all the Ubuntu One startup scripts and considering just purging all the packages and giving up on it :( I really hate that, but since syncdaemon just keeps consuming RAM, rendering my system unusable until I power cycle or manage to get enough cpu cycles to kill the syncdaemon process, I don't have a choice.

I really hope this can be sorted out, because Ubuntu One is a pretty cool tool, and I found it very useful in keeping backups of my development trees, music and other things. I do hope I can use it one day in the future, but for now, I'm left with no other alternative.

[papukaija (papukaija)]

Please open a new bug if you experience this bug in Natty or newer. Thanks in advance.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".