Excessive CPU usage in syncdaemon

[Leo Arias]

This is the master bug for the reported excessive CPU usage, since the other bugs have logfiles and are private.

See bug #902166 for excessive memory usage.

[Marius B. Kotsbak (mariusko)]

Still there in the latest Quantal updates.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stephan Wissel (stw-linux)]

Hunch: it seems to be network related. I have other syncing software, that also starts eating into CPU once the network shows latency challenges. Once network communication terminates (sync complete) CPU load drops

[Felix (felix-akkermans)]

Now over here. Got Dropbox doing syncies and no problem. In the meanwhile the UOne syncdeaming managed to eat an entire core (see attachment). Even Flash is using less CPU while poking it's nose in the background ...

[David Ayers (ayers)]

I can confirm this issue with 12.04 LTS.

My CPU is overheating while the initialization is running. I'm syncing quite a few files (I increased my fs.inotify.max_user_watches to allow me to sync about 150K of files/directories).

During synchronization the CPU temperature is between 80°-85° C but my shortly spike to 93° (as reported by psensors and by /sys/class/thermal/thermal_zone0/temp) often this spike will lead to a shutdown of the system. Of course the whole procedure restart after logging back in. It takes about 40 minutes for the initialization to finish, after which 1.5 GB virtual memory is used by the process of which about 1 GB is resident.

I currently have to avoid any other activity on this system and insure the system is cooled until initialization is finished, especially if Deja-Dup starts running at the same time.

If wonder if a trace would help.
http://docs.python.org/2/library/trace.html
would editing the shebang line in /usr/lib/ubuntuone-client/ubuntuone-syncdaemon work?
Would that help to understand the issue to resolve it?
Or is the issue already well understood and just the way it is?

[John Lenton (chipaca)]

@David,

12.04 already has the "offload queue", that should take care of most memory consumption issues for most people. Your case might be a bit extreme, however.

First, just to confirm, please check whether you have ubuntuone.syncdaemon.offload_queue (do a `dpkg -S offload_queue.py` on a terminal).

Second, exactly how many directories are you watching, such that you needed to tweak proc? The default max_watches is over 500k. Please check whether you're syncing what you think you're syncing:

( find ~/.local/share/ubuntuone/shares/ -type d; u1sdtool --list-folders 2>/dev/null | awk -v ORS="\0" '/subscribed=True/{print substr($0, 64)}' | xargs -0 -Ix find x -type d ) | wc -l

this should output how many directories you're watching (this is what needs to be smaller than max_watches).

About the trace, syncdaemon already has a number of options to profile memory usage. If you want to dig into that, go ahead.

Lastly, you need to get that CPU temperature under control. Anything over 70°C should have you worried, 85+ is bad for your hardware (which is why it shuts down). You should have your CPUs back off before they get that hot (I'm surprised this isn't happening automatically for you).

[David Ayers (ayers)]

@John
Thanks a lot for your reply!

I had to remove 2 ';' to make that find expression work but here is the requested information:

ayers@schiefer:~$ dpkg -S offload_queue.py
python-ubuntuone-client: /usr/share/pyshared/ubuntuone-client/ubuntuone/syncdaemon/offload_queue.py
python-ubuntuone-client: /usr/lib/python2.7/dist-packages/ubuntuone-client/ubuntuone/syncdaemon/offload_queue.py

ayers@schiefer:~$ ( find ~/.local/share/ubuntuone/shares/ -type d; u1sdtool --list-folders 2>/dev/null | awk -v ORS="\0" '/subscribed=True/{print substr($0, 64)}' | xargs -0 -Ix find x -type d ) |wc -l
9736

(This is the default after I had commented my setting to increase the watchers. See bug #761578 )
ayers@schiefer:~$ sudo sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

ayers@schiefer:~$ cat /sys/class/thermal/thermal_zone0/trip_point_0_temp
91000

If the temperature exceeds 91 degrees for more than a really short tome, the system shuts down.

I don't know if there is a sysctl I can set that would slow down the CPU earlier. So I'm not sure how to get this CPU under control. This is a Dell Inspiron 1525 that came with Ubuntu preinstalled and is supported. The only major change is that I have an SSD in it now.

ayers@schiefer:~$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T5450 @ 1.66GHz
stepping : 13
microcode : 0xa3
cpu MHz : 1000.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm
bogomips : 3325.14
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Duo CPU T5450 @ 1.66GHz
stepping : 13
microcode : 0xa3
cpu MHz : 1000.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm
bogomips : 3325.35
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

[David Ayers (ayers)]

Actually forget that comment about removing 2 ';' it seems like a PEBAC while trying to copy and paste.

[John Lenton (chipaca)]

I don't know if it'll help or not (on some dells it works, on some it
doesn't), but you could try `sudo modprobe i8k`. If that works, i8kfan
(in i8kutils). Something like `while true; do i8kfan -2 -2; sleep .1;
done`. I do that with my latitude when doing cpu-intensive work
(because otherwise it gets too hot to handle comfortably). It is
noisy, though.

On Fri, Aug 9, 2013 at 4:14 PM, David Ayers <email address hidden> wrote:
> Actually forget that comment about removing 2 ';' it seems like a PEBAC
> while trying to copy and paste.
>
> --
> You received this bug notification because you are a member of Ubuntu
> One Foundations+ team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/913912
>
> Title:
> Excessive CPU usage in syncdaemon
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntuone-client/+bug/913912/+subscriptions

[EliCoten (launchpad-elicoten)]

I'm not sure if I am affected by the same bug, but I have found that I am getting around 100% CPU usage (most of it is shown as sys by htop) from ubuntuone-syncdeamon.

I'm not even sure what it's doing as I haven't registered for ubuntu one!

This seems to be a fairly recent problem, and doesn't happen immediately on bootup but only when the system has been running for some time.