Interrupted uploads are never retried

Bug #575817 reported by Daniele Napolitano on 2010-05-05
74
This bug affects 15 people
Affects Status Importance Assigned to Milestone
Ubuntu One Client
Status tracked in Trunk
Stable-1-2
High
Unassigned
Trunk
High
Unassigned
ubuntuone-client (Ubuntu)
High
Unassigned
Declined for Precise by dobey
Declined for Quantal by dobey
Declined for Raring by dobey
Declined for Saucy by dobey
Lucid
High
Unassigned

Bug Description

****************
HOW TO REPRODUCE
****************

You'll need to have a broken folder to reproduce this

BY RENAMING THE AFFECTED FOLDER:
1. Computer A has the folder, computer B doesn't
2. Renamed folder in computer A
3. Ubuntu One syncs
4. Computer A has the folder and its components, computer B has the folder without its components

BY MANUALLY COPYING THE AFFECTED FOLDER:
1. Computer A has the folder, computer B doesn't
2. Compressed folder in computer A into a external hard drive
3. Deleted folder in computer A, so no computer has the folder
4. Extracted folder from external hard drive into computer B
5. Ubuntu One syncs
6. Computer B has the folder, computer A doesn't

******************
EXPECTED BEHAVIOUR
******************

- Interrupted uploads to be retried
- Synced folders to have the same contents

**************
REAL BEHAVIOUR
**************

- Interrupted uploads aren't retried
- Sometimes there appear missing contents in synced folders along different devices

***********
WORK-AROUND
***********

REMOVE FOLDER IN THE SERVER:
1. Compress the folder affected by this bug; by right-clicking on it and selecting the "compress..." option, and following steps in the screen
2. Delete the affected folder
3. In one.ubuntu.com, remove the affected folder
4. From the compressed file just created; extract its contents where they were initially located, by right-clicking on it and selecting the "uncompress here..." option
5. Delete the compressed file

****************
RELEVANT DETAILS
****************

- You can easily test if a folder is affected by comparing its number of elements in different devices (righ-click and "properties")
- After the file in a synced folder it is uploaded ('u1sdtool --current-transfers' has shown that all bytes are sent) but server don't assign a hash ID and web interface show this file as 'uploading'. The GUI on Nautilus mark file as synced

**************
TECHNICAL INFO
**************

$ u1sdtool --info=/home/dnax/Ubuntu\ One/gedit-symbol-browser-plugin-bin-debian-i386-0.1.tar.gz
 File: /home/dnax/Ubuntu One/gedit-symbol-browser-plugin-bin-debian-i386-0.1.tar.gz
  info_created: 1273066178.93
  info_is_partial: False
  info_node_id_assigned: 1273066263.74
  is_dir: False
  local_hash: sha1:5cc0f6fb8986905dcd92a798732e47bc95902974
  mdid: dc3e3306-ce19-47d2-8246-7582aa15a2f7
  node_id: 8daff726-0917-44c5-ae56-dc1642e937f4
  path: /home/dnax/Ubuntu One/gedit-symbol-browser-plugin-bin-debian-i386-0.1.tar.gz
  server_hash:
  share_id:
  stat: posix.stat_result(st_mode=33188, st_ino=117868L, st_dev=2066L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=57779L, st_atime=1273066178, st_mtime=1234208141, st_ctime=1273066178)

$ u1sdtool --status
State: QUEUE_MANAGER
    connection: With User With Network
    description: processing queues
    is_connected: True
    is_error: False
    is_online: True
    queues: IDLE

Binary package hint: ubuntuone-client
ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: ubuntuone-client 1.2.1-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.32-21.31-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-21-generic i686
Architecture: i386
Date: Wed May 5 16:12:40 2010
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
PackageArchitecture: all
ProcEnviron:
 LANG=it_IT.utf8
 SHELL=/bin/bash
SourcePackage: ubuntuone-client
UbuntuOneSyncdaemonExceptionsLog: 2010-05-05 15:32:49,806 - ubuntuone.SyncDaemon.ActionQueue - ERROR - Upload share:'' node:marker:dc3e3306-ce19-47d2-8246-7582aa15a2f7 Upload(share_id="''", hash="'sha1:5cc0f6fb8986905dcd92a798732e47bc95902974'", fileobj_factory='<bound method FSKey.open_file of <ubuntuone.syncdaemon.sync.FSKey object at 0xaa6adc4>>', node_id='marker:dc3e3306-ce19-47d2-8246-7582aa15a2f7', crc32='613723678L', previous_hash="''", size='57779') failure INTERNAL_ERROR
UbuntuOneUserSyncdaemonConfig:
 [bandwidth_throttling]
 read_limit = 2097152
 write_limit = 20480
 on = True

Related branches

Daniele Napolitano (dnax88) wrote :
Changed in ubuntuone-client (Ubuntu):
assignee: nobody → Facundo Batista (facundo)
summary: - File uploaded don't receive hash id from server
+ Uploaded file don't receive hash id from server
Changed in ubuntuone-client:
assignee: nobody → Facundo Batista (facundo)
Changed in ubuntuone-client (Ubuntu):
status: New → Confirmed
Changed in ubuntuone-client:
status: New → Confirmed
importance: Undecided → High
tags: added: chicharra u1-lucid-sru
Changed in ubuntuone-client (Ubuntu):
importance: Undecided → Critical
status: Confirmed → In Progress
Roman Yepishev (rye) on 2010-05-06
summary: - Uploaded file don't receive hash id from server
+ Interrupted upload is not retried
dobey (dobey) on 2010-05-06
Changed in ubuntuone-client (Ubuntu):
assignee: Facundo Batista (facundo) → Rodney Dawes (dobey)
importance: Critical → High
milestone: none → lucid-updates
status: In Progress → Triaged
Changed in ubuntuone-client:
status: Confirmed → In Progress

Dobey: remember to merge this branch for the Lucid update:

  https://code.edge.launchpad.net/~facundo/ubuntuone-client/stable--query-no-content-file/+merge/25005

Thanks!!

Changed in ubuntuone-client:
status: In Progress → Fix Committed
HX_unbanned (linards-liepins) wrote :

Isn't Fix Comitted status applicable for ubuntu-one package, too?

Joshua Hoover (joshuahoover) wrote :

HX_unbanned, Fix Committed on the project bug means the bug has been committed to the project's trunk. Since we need to get this fix into Lucid, we need to commit that fix to the Lucid branch, which is separate.

description: updated
Roman Yepishev (rye) wrote :

I am extremely interested in "The GUI on Nautilus mark file as synced"-behavior. Syncdaemon should never signal about the upload as finished if it fails.

Roman Yepishev (rye) wrote :

The item I was extremely interested in - "The GUI on Nautilus mark file as synced" comes from bug #585953.

Accepted ubuntuone-client into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in ubuntuone-client (Ubuntu Lucid):
status: New → Fix Committed
tags: added: verification-needed

I've attempted to reproduce this bug by doing the following:

1. Creating a 600 MB file named "big_file"
2. Copy "big_file" to ~/Ubuntu One
3. Allow upload to begin (checking u1sdtool --current-transfers)
4. Kill the network connection
5. Bring back the network connection (also tried "killall ubuntuone-syncdaemon" as a different test)
6. u1sdtool -c

Result: The file attempts to upload again as expected and eventually makes it to the server.

Making this even harder to test now is the fact that we have updated the server with at least one fix related to this issue. The best method for testing this fix is in the automated tests included in the attached branches. I think this fix is OK. Unless someone who is experiencing this problem currently can test the proposed release, I think we're going to have to trust the automated tests to verify the fix.

Martin Pitt (pitti) wrote :

According to bzr branch state this should already be fixed in maverick.

tags: added: verification-done
removed: verification-needed
Changed in ubuntuone-client (Ubuntu):
status: Triaged → Fix Released
Martin Pitt (pitti) wrote :

ubuntuone-client (1.2.2-0ubuntu1) lucid-proposed; urgency=low

  * New upstream release.
    - Properly handle valid UTF-8 non-ASCII names for UDFs (LP: #557160)
    - Fix nautilus crash when running u1sdtool --subscribe-folder (LP: #570261)
    - Cannot reactivate "File sync" on services tab (LP: #570721)
    - Retry interrupted uploads (LP: #575817)
    - Improve logging at INFO level (LP: #578248)
    - u1sdtool --delete-folder with invalid id hangs (LP: #583412)
    - ubuntuone-syncdaemon crashed with OSError (LP: #452682)
    - ubuntuone-preferences "Got empty result for devices list." (LP: #576263)
  * Rmmove fix-571548.patch and fix-567223.patch; included upstream now.
 -- Rodney Dawes <email address hidden> Wed, 16 Jun 2010 13:42:32 -0400

Changed in ubuntuone-client (Ubuntu Lucid):
status: Fix Committed → Fix Released
tags: added: testcase

I think I'm seeing this in a new installation of 13,04 Raring on an Acer laptop. From my desktop I see a folder (in the Ubuntu One folder) containing 248 items totalling 120 MB, on the laptop I see only the empty folder. Both machines say file sync is up-tp-date.

I did interrupt the the sync on the laptop when I first installed 13.04 (a week ago), because I was nearing my data cap. How do I re-sync?

Changed in ubuntuone-client (Ubuntu Lucid):
importance: Undecided → High

This bug can easily trigger data corruption and lost:

1. Computer A has the file, computer B doesn't.
2. User performs a clean install of Ubuntu in computer A, deleting all his files because he thinks there's a copy on the cloud.
3. Computer A hasn't the file, computer B hasn't the file either.

In my case, this deleted my university work twice. And now I'm unable to sync my work between two computers.

Since this bug can easily cause data corruption and lost , it has a priority of 'critical'.

tags: added: saucy
Changed in ubuntuone-client (Ubuntu):
importance: High → Critical
Changed in ubuntuone-client (Ubuntu Lucid):
importance: High → Critical

I confirm the bug is still there. I have been able to reproduce it reliably, twice everyone, by doing this:

TESTCASE 1 - RENAMING:
1. Computer A has the folder, computer B doesn't
2. Renamed folder in computer A
3. Ubuntu One syncs
4. Computer A has the folder and its components, computer B has the folder without its components

TESTCASE 2 - MANUALLY COPYING:
1. Computer A has the folder, computer B doesn't
2. Compressed folder in computer A into a external hard drive
3. Deleted folder in computer A, so no computer has the folder
4. Extracted folder from external hard drive into computer B
5. Ubuntu One syncs
6. Computer B has the folder, computer A doesn't

Changed in ubuntuone-client (Ubuntu Lucid):
status: Fix Released → Triaged
Changed in ubuntuone-client (Ubuntu):
status: Fix Released → Triaged
Changed in ubuntuone-client (Ubuntu Lucid):
milestone: none → lucid-updates
assignee: nobody → Rodney Dawes (dobey)
summary: - Interrupted upload is not retried
+ Interrupted uploads are never retried

WORK-AROUND - REMOVE FOLDER IN THE SERVER:

1. Compress the folder affected by this bug; by right-clicking on it and selecting the "compress..." option, and following steps in the screen.
2. Delete the affected folder.
3. In one.ubuntu.com, remove the affected folder.
4. From the compressed file just created; extract its contents where they were initially located, by right-clicking on it and selecting the "uncompress here..." option.
5. Delete the compressed file

You can easily test if a folder is affected by comparing its size in different devices (righ-click and "properties"), or with its size in <one.ubuntu.com>.

description: updated
description: updated
description: updated
tags: added: verification-failed
removed: verification-done
tags: added: precise quantal raring

You can reassign this bug to you at any time you re-start working on it.

description: updated
Changed in ubuntuone-client (Ubuntu):
assignee: Rodney Dawes (dobey) → nobody
Changed in ubuntuone-client (Ubuntu Lucid):
assignee: Rodney Dawes (dobey) → nobody
dobey (dobey) wrote :

If you have a problem you can file a new bug. Do not go re-opening old bugs that were fixed.

Changed in ubuntuone-client (Ubuntu):
status: Triaged → Fix Released
Changed in ubuntuone-client (Ubuntu Lucid):
status: Triaged → Fix Released

Since:
- This bug can be reproduced consistently.
- This bug has been reproducible for 3 years and 8 months, in any release and platform.
- There's a triaged by someone else bug report (bug #1112440) that clearly exhibits the same behaviour as this one.

The bug is not fixed yet, except I missed something.

Changed in ubuntuone-client (Ubuntu Lucid):
status: Fix Released → Triaged
Changed in ubuntuone-client (Ubuntu):
status: Fix Released → Triaged

The problem here is how uploads get interrupted is still unknown. This is what needs to be defined in order the bug to be fixable.

@Alberto Salvia Novella (es20490446e) : There can be at least 2 reasons why uploads can be interrupted:

- local host loses power, crashes or reboots
- local host loses Internet access

Anyway, the results are identical as far as the U1 sync is concerned.

@Rodney Dawes (dobey): there are numerous serious bugs regarding U1 sync. I have filed numerous bug reports. None of them seem to have been taken into account so far. Are the developers aware of these numerous problems?

Jean; I'll test to interrupt files as you mention, and see what happens. On the other hand, bugs in Ubuntu One can leanly be stopped by just:

[1] Assigning bugs only just before the assignee will be starting working on them.

So assigned bugs which actually are being worked on (and they're not stopping others to start working on them):
- before: ###-----------------------------------------
- after: ###################--

[2] Start working in bugs in their latest stages of bug management, and only proceed with previous ones when those are finished.

So workforce translated into real bug-fix:
- before: #---------------------------------------------
- after: ###################--

[3] Testing briefly if the just performed action delivered the expected result, and the same for the previous one done by other person.

So defect passed to the latest stages of development:
- before: ###################--
- after: #---------------------------------------------

EXPECTED TAKT TIME:
- Before: ##############################
- After: ##--------------------------------------------------------------------

dobey (dobey) wrote :

@Alberto DO NOT change the status of this bug. If you are experiencing a similar issue, FILE A NEW BUG.

This bug was FIXED. The version it was reported against is EOL, as are Ubuntu 10.04 and 10.10 where it was reported. Updates were shipped with a patch, which is why the bug is marked Fixed Released.

Changed in ubuntuone-client (Ubuntu Lucid):
status: Triaged → Fix Released
Changed in ubuntuone-client (Ubuntu):
status: Triaged → Fix Released
Changed in ubuntuone-client (Ubuntu Lucid):
importance: Critical → High
Changed in ubuntuone-client (Ubuntu):
importance: Critical → High

Since Rodney⚔ has asked to file a new report for this bug in current supported releases, I've done in bug #1270195.

If you're still affected by this flaw, please confirm it by going to <http://tinyurl.com/nnk52dm>.

Thank you 🐱

dobey (dobey) wrote :

@Alberto, this is not a duplicate of the new bug. Just because the symptoms are similar does not mean the bugs are the same. Nobody is affected by *this* bug. It was fixed. Any new bugs exhibiting similar symptoms are totally different bugs.

If anyone thinks they are affected by this bug, please do not change status on bug #1270195. Instead, file a new bug. Everyone with a problem saying "me too" on a single bug does not help fix the bug. All it does is provide a bunch of meaningless comments on a bug.

There is a link for "This bug affects me too" at the top of every bug report on Launchpad. If you feel you must say "me too" then use that link rather than posting a comment. However, complete new bug reports, filed by running `ubuntu-bug ubuntuone-client` are preferred, as they will provide significantly more information to developers.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers