Ubuntuone-process makes hunders of files like /tmp/tmpXXXX, fills hard dirve

[Otto Kekäläinen]

My machine ran out of hard drive space and I noticed that I had 1343 files (22 GB in total size) in /tmp. All of the files were named like tmpXXXXXX, e.g. tmpzwFV1p. Running lsof reviled that the the files were created (and kept open) by the process "ubuntuone".

Background:
I'm running Lucid with all updates and I have a paid 4x20GB U1 account with 20GB+ of stuff and I recently added a folder (~/Kuvat) with 40 GB of files that has uploaded only partially. Account registered to otto at sange.fi.

Analysis:
I restarted my machine (and Linux automatically cleans up the /tmp directory) and documented what happens:

1. When the computer starts, the contents of /tmp is normal.

2. When the computer is on, u1sdtool -s starts with State: LOCAL_RESCAN

3. Then u1sdtool -s shows State: SERVER_RESCAN

4. Then u1sdtool -s shows State: QUEUE_MANAGER

5. At this point just one file is created /tmp/tmpCb3SoN. When the state is QUEUE_MANAGER, uploading should happen and u1sdtool --waiting-content lists 6000+ files, but there is no outgoing network traffic.

6. After a few hours the uploading finally starts. Outgoing network traffic is at level 120-150 KB/s.

7. At the same time more and more files start to appear in /tmp directory. See how it grows by looking at http://otto.kekalainen.net/u1-bugs/tmp-files0.txt, http://otto.kekalainen.net/u1-bugs/tmp-files1.txt, http://otto.kekalainen.net/u1-bugs/tmp-files2.txt and http://otto.kekalainen.net/u1-bugs/tmp-files3.txt.

8. lsof | grep tmp/tmp proves the files are opened by process ubuntuone. See how it grows in http://otto.kekalainen.net/u1-bugs/lsof01.txt and http://otto.kekalainen.net/u1-bugs/lsof02.txt

9. Finally the filesystem gets full. Listing http://otto.kekalainen.net/u1-bugs/filesystem-full.txt reveals also the file sizes. They seem to match the file size of files in ~/Kuvat.

10. Now u1sdtool --waiting-content shows only one file at the time:
otto@shuttle:/tmp$ u1sdtool --waiting-content
operation='Upload' node_id='81ca5875-91a3-4b49-827f-48aad081f1fb' share_id='492ae22d-006c-4044-b457-e0a6b6a9ceca' path='/home/otto/Kuvat/2010-marraskuu/IMG_3590.JPG'
otto@shuttle:/tmp$ u1sdtool --waiting-content
operation='Upload' node_id='6c1e799d-4584-4164-9c62-262053688138' share_id='492ae22d-006c-4044-b457-e0a6b6a9ceca' path='/home/otto/Kuvat/Sekalaista/Joulu 2008/00115.jpg'
otto@shuttle:/tmp$ u1sdtool --waiting-content
operation='Upload' node_id='ae9f7fd2-bff6-48d9-8508-0a8e6e0b831b' share_id='492ae22d-006c-4044-b457-e0a6b6a9ceca' path='/home/otto/Kuvat/Sekalaista/Pariisi 2009/IMG_8016.JPG'

11. However when I open at one.ubuntu.com > My Storage / ~/Kuvat / Sekalaista / Joulu 2008, there is no file 00115.jpg

12. Network traffic shows that uploading is happening.

13. Some other programs start to bug when the filesystem is full. Computer becomes unusable.

I can reproduce this on my machine every time I restart it and wait a few hours.

[Otto Kekäläinen (otto)]

apport information

[Otto Kekäläinen (otto)]

apport information

[Otto Kekäläinen (otto)]

apport information

[Otto Kekäläinen (otto)]

apport information

[Otto Kekäläinen (otto)]

apport information

[Otto Kekäläinen (otto)]

Added .cache/ubuntuone/log/syncdaemon.log while running with the problem present (/tmp is now full).

[Otto Kekäläinen (otto)]

Issue was discussed at #ubuntuone. Log show a lot of TRY_AGAIN. Now we need to find out why is does TRY_AGAIN or find a way to handle deflated files better so they don't eat up /tmp:

[13:44] <rye> facundobatista, hmm, is it possible that upon receiving TRY_AGAIN on upload (bug #702309) the client does not close the filehandle but moves to another content queue item thus accumulating the deflated files?
[13:50] <facundobatista> rye, on one hand, the deflated file is created when the Upload starts, and removed if the Upload fails, or finish successfully
[13:51] <facundobatista> TRY_AGAIN is retryable, so the deflated file is not removed (as will be used again)

[Otto Kekäläinen (otto)]

I removed 50 % of the contents in ~/Kuvat, including a big 12 GB video file, and now the upload seems to work normally. There is only one temporary file in /tmp/tmpxxxxxx at the time and the upload que is shrinking:

otto@shuttle:~$ u1sdtool --waiting-content | wc -l
6172
otto@shuttle:~$ u1sdtool --waiting-content | wc -l
6169
otto@shuttle:~$ ls /tmp/
00108.jpg keyring-Y5T4iI ssh-qcXdRQ1269
context ksocket-otto tmp5FLe85
gnome-system-monitor.otto.4151304140 orbit-otto virtual-otto.dYBlRk
kde-otto pulse-Z4BJvkDL3HRh
otto@shuttle:~$ u1sdtool --current-transfers
Current uploads:
  path: /home/otto/Kuvat/Kesä 2009/1.IMG_2839.JPG
    deflated size: 2005469
    bytes written: 2005469
Current downloads: 0

Conclusion: the Ubuntu One client in Lucid is unable to handle big files or too many files correctly.

[Otto Kekäläinen (otto)]

Now a week, 6000 files and 20 GB later, all that is left untransferred is two >10 GB video files. Apparently the transfer does not even start: u1sdtool --current-transfers toggles between "bytes written: 0" and "bytes written: 195708".

Conclusion: Ubuntu One (at least in Lucid) is unable to upload large files (in my case >10 GB).

The large file that cannot be uploaded "fills" the uploading que and thus the /tmp directory was filled with deflated temporary files in the que that did not get past the big file.

Solution 1: Add to Ubuntu One marketing a warning, that there is a limit how big the file can be. Make the client to give a error when the file size is too big for upload (or when the uploading of a file fails hundreds of times in a row).

Solution 2: Make Ubuntu One support files of any size by supporting some mechanism of partial/interrupted uploads.

[Otto Kekäläinen (otto)]

From #ubuntuone:
[13:38] <otto__> I filed #702309 and now got to the root of it: U1 does not support uploading files in class 10 GB
[13:39] <rye> otto__, hm, max file size is 5Gb as far as I remember
[13:39] <rye> and i suppose this is written nowhere and client is not notifying about that :-/

[Jeffrey Patrick Lui (punong-bisyonaryo)]

I can confirm that this also happens in Natty.

[Jeffrey Patrick Lui (punong-bisyonaryo)]

I've seen files as small as 13.8MB get repeatedly created in /tmp