ubuntuone-syncdaemon consumes excessive memory

[Chris Halse Rogers]

During the initial synchronisation process it seems that ubuntuone-syncdaemon consumes excessive memory.

My UbuntuOne directory contains ~200Mb worth of data in ~20,000 files.

Top output is:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9707 chris 20 0 1565m 1.4g 2844 R 99 48.9 35:06.44 ubuntuone-syncd

1.4GiB RES seems excessive :)

[Chris Halse Rogers (raof)]

.home.chris..cache.ubuntuone.log.syncdaemon.exceptions.log: 2009-05-26 18:42:21,416 - ubuntuone.SyncDaemon.ActionQueue - ERROR - ListDir share:'' node:'22216fc3-fe5a-46aa-901c-ef3a7aeb448b' ListDir(node="'22216fc3-fe5a-46aa-901c-ef3a7aeb448b'", fileobj_factory='<function <lambda> at 0x3587a28>', share="''", server_hash="'sha1:989cedf3fff9c45ebc6fa4fff3544882a7915706'") failure unable to build fileobj (file went away?) so aborting the download.
Architecture: amd64
DistroRelease: Ubuntu 9.04
Package: ubuntuone-client 0.90.1-0+r25-2~ubuntu.9.04
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 SHELL=/bin/zsh
 LANG=en_AU.UTF-8
UbuntuoneClientPackages:
 ubuntuone-client 0.90.1-0+r25-2~ubuntu.9.04
 ubuntuone-storage-protocol 0.90.0-0+r34-0~ubuntu.9.04
 ubuntuone-ppa N/A
Uname: Linux 2.6.30-1-mmiotrace x86_64
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare

[Guillermo Gonzalez (verterok)]

Hi Chris,
Thanks for reporting this issue.
Could you describe the steps to reproduce this? (this is mainly to help me debug it)
e.g:
 1) started ubuntuone client,
 2) copied the 20.000 files inside My Files (or maybe you copied 5000 files and then other 5000, etc)

Thanks a lot!

[Facundo Batista (facundo)]

Guillermo:

Note, about the files, that they're inside a Dropbox directory, and (at least in the cases I looked) they keep changing, creating and disappearing all the time.

[Guillermo Gonzalez (verterok)]

Facundo,
Thanks for pointing that out :)
I'll try to write a test to reproduce the behaviour and profile the
memory usage.

Chris: are you using UbuntuOne client *and* Dorpbox at the same time
controlling the same directory?

On Tue, May 26, 2009 at 3:07 PM, Facundo Batista <email address hidden> wrote:
> Guillermo:
>
> Note, about the files, that they're inside a Dropbox directory, and (at
> least in the cases I looked) they keep changing, creating and
> disappearing all the time.
>
> --
> ubuntuone-syncdaemon consumes excessive memory
> https://bugs.launchpad.net/bugs/380533
> You received this bug notification because you are a member of Ubuntu
> One hackers, which is subscribed to Ubuntu One Client.
>
> Status in Ubuntu One Client: New
>
> Bug description:
> During the initial synchronisation process it seems that ubuntuone-syncdaemon consumes excessive memory.
>
> My UbuntuOne directory contains ~200Mb worth of data in ~20,000 files.
>
> Top output is:
> PID     USER      PR  NI  VIRT     RES   SHR  S  %CPU %MEM    TIME+      COMMAND
> 9707  chris       20   0   1565m 1.4g  2844 R   99      48.9       35:06.44  ubuntuone-syncd
>
> 1.4GiB RES seems excessive :)
>

--
Guillermo Gonzalez
<http://launchpad.net/~verterok>

[Chris Halse Rogers (raof)]

I just dropped my "Documents" directory (containing ~20,000 files)
into "My Files" in the UbuntuOne directory. This contained a Dropbox
controlled directory, but Dropbox isn't (or shouldn't be, unless
ubuntuone does some strange hooking into gvfs copy operations or
something) controlling ~/Ubuntu One/My Files/Documents/Dropbox, it's
watching ~/Documents/Dropbox.

So, the test case would indeed be:
1) Start the ubuntuone client
2) Copy a directory containing a directory tree with 43,850 (as
counted with ls -R -1 | wc -l) odd files into My Files.

[Guillermo Gonzalez (verterok)]

Hi Chris,

On Wed, May 27, 2009 at 7:13 AM, Chris Halse Rogers <email address hidden> wrote:
> I just dropped my "Documents" directory (containing ~20,000 files)
> into "My Files" in the UbuntuOne directory.  This contained a Dropbox
> controlled directory, but Dropbox isn't (or shouldn't be, unless
> ubuntuone does some strange hooking into gvfs copy operations or
> something) controlling ~/Ubuntu One/My Files/Documents/Dropbox, it's
> watching ~/Documents/Dropbox.
>
> So, the test case would indeed be:
> 1) Start the ubuntuone client
> 2) Copy a directory containing a directory tree with 43,850 (as
> counted with ls -R -1 | wc -l) odd files into My Files.
>

Thanks for the detailed description, this would help us to reproduce the memory consumption.

> --
> ubuntuone-syncdaemon consumes excessive memory
> https://bugs.launchpad.net/bugs/380533
> You received this bug notification because you are a member of Ubuntu
> One hackers, which is subscribed to Ubuntu One Client.
>
> Status in Ubuntu One Client: New
>
> Bug description:
> During the initial synchronisation process it seems that ubuntuone-syncdaemon consumes excessive memory.
>
> My UbuntuOne directory contains ~200Mb worth of data in ~20,000 files.
>
> Top output is:
> PID     USER      PR  NI  VIRT     RES   SHR  S  %CPU %MEM    TIME+      COMMAND
> 9707  chris       20   0   1565m 1.4g  2844 R   99      48.9       35:06.44  ubuntuone-syncd
>
> 1.4GiB RES seems excessive :)
>

--
Guillermo Gonzalez
<http://launchpad.net/~verterok>

[frankip (frank-priscoglio)]

I have also experienced the exact same problem. Started client, copied 11814 files into the my files directory. It is now consuming 2.7GiB of memory and is swapping like crazy.

I am running 9.04 64Bit with 4G of RAM.

[Jake Rayson (growdigital)]

Same issue here, 9.04 32bit 2GB RAM, it consumes 80%+ of the memory!

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 3321 dogsthat 20 0 1899m 1.6g 1592 D 15 80.9 73:41.06 ubuntuone-syncd

[Mark Schewe (quorr)]

I have the same problem.
I copied about 5,000 files ~ 60 MB ( very small files - various git repositories )
After a while the sync client starts to use more and more memory, until the system gets really slow.... at the end it uses about 1,6GB of 2GB RAM.

I am using Ubuntu 9.04 32bit with all current updates.

[Redge (redgetrek)]

Same here with 13268 files at ~ 500 MB. After a while it runs up to >80% of 8 (!) GB RAM.

[Matt Griffin (mattgriffin)]

Raising the importance of this bug to Critical.

[RUSHER81572 (phillip-tribble)]

I ddid not sync anything and I found 1GB of my 4GB total being used by ubuntusync-daemon. I hope this helps.

[Mark Combellack (combellack)]

I am having the same problem too with excessive memory usage. Currently the ubuntuone-syncdaemon is consuming 2.9GB of memory. It is in the scanning state.

My UbuntuOne directory contains a copy of my files from my DropBox folder (cp -r Dropbox Ubuntu\ One/My\ Files). The stats are:
  Total file size = 188MB
  Total file count = 13961

I am using Ubuntu 9.04 64-bit and using ubuntuone-client version 0.90.3+r64-0ubuntu1~ppa~jaunty

[philippem (michel-philippe)]

I have the same problem.
Use of memory : 1.9 Go (RAM : 2Go)
Use of swap : 50 %

[Timur (timka.org)]

I have an empty My Files directory and ubuntuone-sycndaemon is doing nothing but its RSS is 159m after the startup, twice as big as my firefox-3.5 with two tabs.

[Guillermo Gonzalez (verterok)]

(A brief status update)

After some digging, and some fixes that improve things a bit but don not fix this issues (yet), I found that the local rescan is the major offender regarding memory consumption.

During my tests I found that deep directory hierarchies are one of the problems

Steps to reproduce :

With the syncdaemon turned off execute the script:

letters="a b c"; mkdir -p test_1; cd test_1; (set -e; for i in $letters; do mkdir $i; for j in $letters; do mkdir $i/$j; for k in $letters; do mkdir $i/$j/$k ; for l in $letters; do mkdir $i/$j/$k/$l ; for m in $letters; do mkdir $i/$j/$k/$l/$m ;cp ~/tmp/1k.txt $i/$j/$k/$l/$m/$i$j$k$l$m ;echo $i/$j/$l/$m/$i$j$k$l$m;done;done;done;done;done;) cd ..;

This will create only 243 files, and 364 directories, but the files are 5 level down in the hierarchy (including root) e.g: test_1/b/b/b/b/b/bbbbb

The memory consumption (reported by top) once syncdaemon stops working is:
VIRT: 772MB, RES: 622MB

To confirm that the problem is with deep directory hierarchies, I created a flat (but large) hierarchy on disk:
values=`seq 5000`; mkdir -p test_2; cd test_2; mkdir a;(set -e; for i in $values; do cp ~/tmp/1k.txt a/$i ; echo a/$i; done;); cd ..;

This creates the 'test_2/a" directory with 5000 files (1k size) in it.

Now, the memory consumption reported by top is:
VIRT: 219MB, RES: 67MB

as a reference: the syncdaemon consumes ~45MB just after the startup (with a clean root)

[A. Bram Neijt (bneijt)]

This bug effects me two, I'm on 64bit with all the updates of today installed.
ubuntuone-client 0.90.4-0ubuntu1

To reporoduce this problem, I extracted the Linux kernel source into the My Files directory:
cd /tmp
wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.30.2.tar.bz2
cd ~/Ubuntu\ One/My\ Files/
tar -xjf /tmp/linux-2.6.30.2.tar.bz2

I then let UbuntuOne turn the logo a bit and then reboot. After reboot, I my computer started slowing down and swapping, at which point I killed and then kill -9 the ubuntuone-syncdeamon program. The second kill was needed as it was uniterruptible waiting on channel sync_page.

[A. Bram Neijt (bneijt)]

I hope this log also helps.

I poked around a bit and saw that my queue for the shares grew to above the 500 and they seem to contain a listing of all directories, according to:
self._queue.insert(0, (all_share_dirs, share.path, share.path))
So it may be that 500 times a full directory listing in all_share_dirs is taking up space??

Good luck!

[Guillermo Gonzalez (verterok)]

Hi bneijt,

Thanks for the followup and the logs.

> So it may be that 500 times a full directory listing in all_share_dirs is taking up space??

Indeed, I'll give a closer look to that bits.

Thanks!

[Guillermo Gonzalez (verterok)]

A fix is available in the nightlies ppa (https://edge.launchpad.net/~ubuntuone/+archive/nightlies).
I tested this version with the kernel source tree as described by bneijt, and the results are quite good compared to the previous version:
The local rescan took 20min and the memory consumption around 200MB

P.S: this time don't include uploading

[cande (cande)]

Similar issue, running Ubuntu 11.10 latest beta with 4 Gb of memory, and put one big directory form dropbox to ubuntuone, its consuming about all of my memory.

[bluenova (bluenova)]

I 2nd Cande the issue still remains in Oneiric.

[Dovev Hefetz (dovevh)]

I am running ubuntu one on Windows 7 on a new laptop with 4 GB RAM syncing My Documents folder and am seeing similar results:
Memory consumption: 1.45 GB
My Documents contains 785 MB, 7550 Files, 2,823 Folders (only 11 top level folders).

I see that a fixed was released according to the bug status - was that for the windows version also?

[Jesse B (jayesbee)]

I will 3rd that the issue does remain in the latest 11.10 with the 3.1.4 kernel..

Linux 3.1.4-030104-generic #201111281851 SMP Mon Nov 28 23:52:23 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

After booting.. I have no choice but to immediately run terminal and kill the ubuntuone-syncdaemon process that will baloon to 2.9gb

I am only syncing 2 hidden directories in my home folder.. one for PAC manager (~/.pac) and MySQL Workbench (~/.mysql) so I can sync these across desktops between work and home.

[Alaak (alaak-gmx)]

I can also confirm that the issue still exists for 11.10. It happens for me if I add some directories containing some Gigabytes of data to the synchronization. Memory then runs full within a timeframe of one hour and the client reports that synchronization of Files is started up. The german message is: "Abgleichen von Dateien wird gestartet ..."