Missing real sync feature

[Robin Sheat]

This is more a wishlist than anything.

Anyway, I use U1 files for syncing documents between my tablet and the desktop, so I can work on them whereever (in this case, using freemind and thinkingspace, it's almost completely seamless.) However, when I've made changes on the phone version, I notice that it doesn't seem to automatically upload the file; if I manually add the file, then it works fine. It'd be nice to have the phone keep the file in sync too.

(Or, am I missing something and it does actually do this, after some delay or something maybe?)

[Michał Karnicki (karni)]

Correct, there is no sync apart from the auto-upload feature. We will be looking into adding this feature, thanks!

[Dan Trevino (dantrevino)]

this appears to manifest itself as not being able to "add" files as well. or should that be a separate bug? i click on the (+) icon in the corner, add a file, and it never shows up. I've even moved files manually to the u1 folder and still nothing.

[Robin Sheat (eythian)]

I can always add files, and they overwrite the existing one (which is what I want) so it sounds like yours may be a different issue.

[Michał Karnicki (karni)]

@Dan
- Are you using OI File Manager perhaps? Try AndExplorer or Astro, please, just to see if it works.
- If you move files manually to u1 folder, nothing happens - because Ubuntu One Files does not support sync, other than auto-uploading pictures you take with camera. Please let me know where we can make it clearer U1F does not feature sync, as so many users expect it to. We've been trying to make it very clear, but this kind of feature will be coming in 2-3 months from now.

As far as this bug is concerned, this is strictly a sync-feature - when we have that, I believe we'll let the users choose if we should sync file deletions as well. Some users expect to delete stuff from their phone, as they know it's in already in U1 (whilst that would remove it from U1 as well).

[Ichbinich (chris-walter)]

I just encountered this bug in the most annoying way - I altered files on my android phone last month or so. After this I altered the file again many times on the computer without noticing that the changes I made on my phone disappeared. Today I searched for the changes I made on the phone and found out that they are gone.

After some searching I found this bug report, which is on importance "wishlist". Are you serious? What the h*** is a syncing client worth if its not syncing? It seems that I cant even upload an altered file to the cloud from my phone. And this essential feature is just a "wishlist" one? This should be top priority....

A cloud app that is not syncing... I really cant believe this is true....

[Michał Karnicki (karni)]

> What the h*** is a syncing client worth if its not syncing?

Hi Chris,
I am sorry you lost your changes. Ubuntu One Files is still just a client of Ubuntu One storage (similarily to the iOS version), it is does not yet sync. In the automatic manner, we only auto-upload photographs from the device, if the user wishes so. Please note you will not find any hint in descriptions of Ubuntu One Files, that it is a "sync client". Until it is, we make sure we are very clear about that.

On the other hand, we do have sync in the near future and once it's done, we hope you will give it another try.

PS To reupload a file from the device, you need any file browser and just add it as any other custom file (plus sign on the action bar). Uncomfortable, I know. We'll make it better.

[Michał Karnicki (karni)]

Also, please note we strive to ship sync as soon as we can. We have to prioritize features (such as OOM upon initial auto upload) or depend on APIs - in case of sync, the API that will enable much easier sync is on the way.

[Jeffrey Patrick Lui (punong-bisyonaryo)]

> We've been trying to make it very clear, but this kind of feature will be coming in 2-3 months from now.

Hi Michael! I'm so excited for this. So i guess this feature will ship out at about the same time as 12.04?

[Michał Karnicki (karni)]

Hello Punong! We are finishing the API part that will enable us to do sync much smarter/lighter. I hope we can integrate this into the app by the end of April - I can't promise though. Stay tuned!

[Alex Bluk (alex-bluk)]

Hi Michael,

I just got a new Android device and was willing to use Ubuntu One for syncing. I was also disappointed to see that it does not upload modifs , but I'm glad to see you're working on it...

Just to let you know another user is looking forward to use your work ! Good luck with the development.

[axel (x.)]

Hi Michael,

i would really lke to see a real sync being available on the android u1 app.
google drive can't do this either (yet) but i really would prefer to keep using u1.
so... any news on the progress of (automatic) folder/file sync for the android u1-app?
anyway, thanks for your efforts in explaining the unavailablility of sync, i was about to post another bug but luckily found this one...

regards,
axel

[Michał Karnicki (karni)]

Hi Axel,

At the moment I am the only one working on our Android apps (and related libraries), and I have to balance feature requests and Canonical roadmap. I know I thought we'd deliver this sooner, but the sync is still before us on the roadmap, and I was recently focusing more on backend libraries than app features. Hopefully this will change soon.

[Selene ToyKeeper (toykeeper)]

I would love to see this feature added too, though in my case I'm more interested in auto-sync downloads than I am in uploads.

I'd like to keep an updated copy of some desktop files on my phone for reference. The first thing I tried was syncing a summary of my task list to the phone so it can display in a widget in a launcher screen. (The "Word Widget" app works for this, and is the only way I've found to display a self-updating text file in a widget) However, for now, I can implement this with scp and a few lines of shell code.

[Michał Karnicki (karni)]

Hi Selene,

FWIW, this part is easier to implement. It is harder to detect local changes, say, when a user edits a text file or removes red eye outside of U1F. Because of that, we didn't want to deliver partial solution, because when we say "sync" users will expect more or less what the desktop client does. Keep you in sync. Therefore, we will implement sync-down only when we have rock solid sync-up plan. We have this on our roadmap.

[iatll (iamthelamplighter)]

Michal,

The app currently shows the last modified date of each file (This is the last date at which the file was "synced" in the cloud, right?). When the app refreshes the file list, could you not grab the File.lastModified() value to determine if a local change has been made recently? If the last modified date provided by the cloud is not accurate enough, couldn't you just generate and store your own "last modified" timestamp whenever a new version of a given file is downloaded?

In other words, Android's SDK provides a simple way of determining when a file was modified, and comparing that value with the cached/stored "last downloaded/uploaded" value should be enough to detect all local changes.

Please forgive me if I'm misunderstanding the technical problems that detecting local changes presents. I'm an Android developer as well, and if I can do something to help work on this bug I'd be happy to help. If you can point me to where the app stores file information, I'm sure I could implement this myself and submit a revision to you to approve.

[Michał Karnicki (karni)]

Hello iatll :)

The dates visible near files are last modification date. So if you upload a file from a computer, you'll see "just now" on the phone. If you modify the file on the computer and some time passes, you'll see "last modified <the_date>". Basically, whenever a file is modified on any device, and this change makes it to the servers (the cloud), you'd see that last modification date on the list.

The last modified timestamp from the server is accurate and useful if we want to sync the changes down to the device. Question is - which changes do we want to sync, because surely not all. Meaning, it has to be selective, because your Ubuntu One directory can be much more gigabytes than what you have on the mobile device. We have in plans to keep in sync whatever has been downloaded, but I'm not 100% yet convinced this is the right way to go, because it would possibly require waste of battery/resources when rescanning file modification dates in which you are not interested in. But - this part in general is much easier.

When it comes to syncing up files, we do not know when files change. When you edit a text note, or remove red eye from a photo, or change your password in keepassx store. Besides Android 4.0 which has MediaStore.Files, so we could watch that to get notified of file changes. However, for any older Android version (2.3.3 is still the major chunk of the market) we'd have to either to periodic selective rescans (most probable scenario) of force the user to open these files (with their favorite apps) from within U1F if they expect local changes to sync up.

Least pleasing solution (but it'd still work) would be to detect local changes only when browsing into a given directory. Once you see it, and cache gets refreshed, it would trigger uploads/downloads of files accordingly. FYI This cache refresh is the only point in U1F where local file copy is removed from the device in an automatic manner *if* the server has returned a newer file version, so that you don't use the old one locally. (Instead of syncing it down without confirmation, we've chosen to delete the local copy of the mobile device instead, so that when you tap it, you'll download the newest version.)

Both sync up and down need to handle conflicts properly (possibly moving the modified version of the file that conflicts to file_name.u1conflict).

Lastly, there's some design work in progress regarding the UI, so we'll be utilizing that when it comes to implementing sync. Also, this feature is so critical (when we finally implement it) that I will probably take it up myself, but I am always happy to accept contributions or hack together with folks on U1F! bzr branch lp:ubuntuone-android-files to get the code. There's com.ubuntuone.android.files.provider.MetaProvider which provides access to all cached metadata from the server. Most probably MetaService would play an important role marking files as dirty for sync or such, and we could use Android sync framework to do the heavy work.

[iatll (iamthelamplighter)]

Hi Michal, thanks for getting back to me with so much information!

Have you considered using a single instance of android.os.FileObserver (Available since API level 1) inside a constantly running Android service? It fires whenever -any- process views or modifies a file or directory. So one possible solution would be to have FileObserver watch for any change in the parent Ubuntu One directory (on the sdcard/device). When a modify event occurs, this function can then check every file in the Ubuntu One directory for its last modified timestamp using java.io.File.lastModified() and compare this value to the cached timestamp. If it's newer, we upload. This way, we're not burning CPU & battery by constantly checking last modified dates.

If for whatever reason there are problems associated with using this method, I think it is critical to provide -some- form of upload. The most basic case would be to have a last modified scan occur when the ubuntu one app is opened, for uploads to occur then. This way, we're not committing to "full sync" yet (since we're not really ready for it) but we're still providing some method for users to upload device->computer.

I completely agree with your reservations about always running sync-down. I don't have a data plan on my cell phone and most of the files I keep in my Ubuntu One folder on the computer aren't needed on my phone, so I actually prefer the way downloads work now. FileObserver may be of help again here though - it will also fire an event when a file is accessed or opened from any process. We can catch this event and then check for downloads at this time. Unfortunately by the time the check completes the user would most likely already have it open in whatever app the request originated from, so we could simply display a notification that a newer version exists.

I understand that this feature is very important and that you'll probably want to put it together yourself. In the meantime, I'm going to try to try running some tests on FileObserver to see if it'll work. I've downloaded the source code from lp:ubuntuone-android-files, but I'm not able to compile it at the moment - I think I'm missing some external jars. I'll get back to you if I'm able to make any progress, but let me know what you think about this possible solution. I expect that I will be able to figure out some scheme to detect when files are changed, but I will need to read up more on the ubuntu one/android sync api to get any uploads started.

[Michał Karnicki (karni)]

Hello!

Thank you for the time you're dedicating to this :)

As far as FileObserver is concerned, I have really bad news for you. The low level inotify on Android has been modified to notify about parent and immediate children events ONLY, it's non recursive. Meaning, if you have Ubutnu One/foo/bar.txt and register a FileObserver on Ubuntu One directory, you won't get any notification if there's an event for bar.txt. For that reason, we would have to register a FileObserver for every parent directory of files you are interested in tracking for upload. Something like selective sync up, where you tell U1F "Make sure files from folder A, C and Z are uploaded whenever I modify them. Ignore the rest.". I haven't yet looked at memory footprint, but I think registering multitude of FileObservers may be very fragile. It may turn out a better solution than periodic scan, though. I've made some initial research into using FileObservers, but I can't find the branch right now.

"The most basic case would be to have a last modified scan occur when the ubuntu one app is opened"
Say, rescan all files which are marked as cached by U1F? And another question, do you think there exists an edge case where a user edits a large file locally, but doesn't want to re-upload that to U1 automatically? (For the second question, I don't think there is, which would be good.)

I'm very happy if you'd like to dedicate some time to look into possible FileObserver use.

Building U1F is super easy, make sure you read HACKING.txt :) However, you may find a problem with a thumbnail related call - fell free to contact me directly at michal.karnicki at canonical.com or just comment out that line.

[Alejandro J. Cura (alecu)]

On Tue, Sep 4, 2012 at 11:11 AM, Michał Karnicki
<email address hidden> wrote:
> As far as FileObserver is concerned, I have really bad news for you. The
> low level inotify on Android has been modified to notify about parent
> and immediate children events ONLY, it's non recursive.

This is a limitation on the Linux kernel: all of inotify is non-recursive.
On our desktop application we have to work around this by adding a
Watch per folder, and this surely takes more ram and cpu, which are
scarcer on a mobile device.
Plus the number of watches allowed per user process is limited, so be
sure to have that in mind when working with large trees of folders.

[Michał Karnicki (karni)]

Good point, alecu. I was wrong. inotify is simply non-recursive.

For those interested, from inotify manual:

   Limitations and caveats
       Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for
       large directory trees.

       The inotify API provides no information about the user or process that triggered the inotify event.

       Note that the event queue can overflow. In this case, events are lost. Robust applications should handle the possibility of lost events gracefully.

       The inotify API identifies affected files by filename. However, by the time an application processes an inotify event, the filename may already have been deleted or renamed.

       If monitoring an entire directory subtree, and a new subdirectory is created in that tree, be aware that by the time you create a watch for the new subdirectory, new files may
       already have been created in the subdirectory. Therefore, you may want to scan the contents of the subdirectory immediately after adding the watch.

[iatll (iamthelamplighter)]

Wow, ok. Thanks for the information! Looks like inotify isn't the best way to go..

I'm going to try writing a simple Service that periodically does a full scan of every folder for a file that's been modified (and that we've cached). I think we can use some "smart" tactics based on user behaviour to greatly decrease the cpu/memory load of this scan. For example:

I am hypothesising that users would readily accept a 2-5 second delay between saving a file and having it uploaded. It will be at least a couple of seconds from the time that the user first saves the document to the time that he/she will be able to open u1 to check if the file is being uploaded. Preliminary tests that I've run (on an emulator, running on a laptop - so it's slow) show that it takes ~70ms to do a full scan of 100 files (tests also show this increases linearly - it takes ~700ms to do 1000 files). Clearly this is unacceptable to run all the time for large amounts of files, but perhaps we can decrease it even more...

Since the user is on a phone or tablet device, the user will finish his editing session by locking/sleeping the device. This is a perfect time to scan! We can wait ~10 seconds after the screen turns off, grab a wakelock, scan & upload, then release the wakelock and let the device sleep as normal. Since we know the screen is off, we can completely stop scanning because we know the user won't be editing anything. In fact, we may even be able to get away with increasing the time between automatic scans to 1-2 minutes because we will -always- catch when a new file has been edited in time to sync it up.

Of course, there are lots of edge cases with large assumptions like this - what if the user likes to turn off wifi manually after editing a file, before we have a chance to upload it? What if the user shuts off the device instead of sleeping it? However, if these cases are sufficiently rare, (and who turns off their wifi when expecting something to sync...) we can solve it by ignoring it and showing warnings instead.

So in theory, we can get away with doing computationally intensive full scans by doing them infrequently and selectively. Since the user's really only changing files when the screen is on, we can stop scanning when its turned off, saving battery and cpu.

Regarding your question
"And another question, do you think there exists an edge case where a user edits a large file locally, but doesn't want to re-upload that to U1 automatically?"
We cannot rule out the possibility that this case exists, but I think that in almost every case, the user will not want to re-upload this file automatically. Actually, we can get around this by automatically marking large files as do-not-auto-upload and displaying a warning (or letting them set the threshold for themselves!). And regarding the full scan, we can also allow users to set which folders to scan, decreasing the amount of files we'd have to check.

[axel (x.)]

hi iatll,

great to see you stepping up with new ideas fuelling possible slutions for this bug! altough lacking some essential adroid development skills i pretty well understand the complexity of finding a good strategy for triggering sync-scans.

reading your recent descussion i became a little worried about some assumptions regarding file-changes since from the experience of my own use-cases found some conflicts/ideas that might be taken into account:

- sync-relevant file-changes only occur during screen-on:
files could also be altered/added/removed externally (e.g. usb-mount, ssh) where screen is not relevant at all. or other services that produce/alter data like gps-loggers (i use this quite often) or external peripherie like IOIO boards with sensors or logs. if no scans at all are run during screen-off times the u1-client would miss these changes until next screen-on, right?
i would suggest to widen the scan-intervals during screen-off or even giving the user a set of values to choose from for screen-on and screen-off with a hint that slower intervals provide better battery runtimes like seen in other applications.

-trigger sync after screen-off event:
in my current configuration, wifi is switched off automatically shortly after the screen is turned off, switching back to slower and limited 2g/3g (if enabled).

-changes of bigger files:
i expected that u1 uses some rsync variant for synchronization so only changed parts of files are actually transferred which makes me wonder why bigger files should not be included in sync. one usecase coming to my mind is syncing local git-repos (in my case i carry them on my phone and copy them to various (offline) machines if i need to change something and move everything back after i am done).

of course, if a single scan of 100 files takes around 700ms and this value scales almost linear a certain limit of observable files is in sight and the u1 client should imho try use as little resources as possible.
an idea here could be to stretch the scanning process a bit (leading to longer times until a scan is finished in total) by dividing the complete list of folders/files to scan into smaller packs that are scanned with some idling in between.
so if lets say only 50 files are scanned at once and then a pause of some seconds is inserted before the next pack of up to 50 files is scanned the system would behave responsive and there are chances that some changes are instantly detected and marked for synchronization. then, if all folders/files that are to be scanned have been checked the client could lookup if the next wave of sync-checking is already due and contine from the beginning or sleep until whenever the next run is due.
this strategy would provide the benefit of being very responsive in cases where only few files need to be synched while keeping the system-load low also for cases where plenty files need to be synched with the price of a slower change detection in total.

regards, x

[iatll (iamthelamplighter)]

Hi axel,

It's great to have your input, thanks! It is very useful to hear about all these use cases, so I can consider how to handle them properly.

If files are altered externally, would I be correct in assuming that the sdcard would be dismounted first? If so, then we can also watch for this and handle appropriately - we can't download to the sdcard if it isn't attached in any case. That still leaves the other data logging services though, you are correct. I think your suggestion to widen the scan interval is best here, but I believe that by default the action should be to not upload and stop syncing until the screen goes on again. Waking the phone from sleep/deep sleep, even for a second, is -extremely- expensive. However, because of the use case you mentioned, I'll work on supporting the option of allowing the service to check while the screen is off.

Wifi switching off immediately shouldn't be a problem - it's possible to obtain a wifi wake lock, and (correct me if I'm wrong) I believe the current app does grab one while it uploads/downloads. The question is - is it ok if the user has set his/her wifi to switch of, say, 2 seconds after screen off and u1 sync service delays this to 4? (or more - if a file starts uploading and takes several seconds to complete)

Small correction, it takes around 700ms for a scan of 1000 files. I'm not quite sure, but I think the app currently has you download files individually, so I don't see most users really syncing anything more than 20-30 files. However, to support this use case, yes - I was thinking of spreading out the scan action as well.

[Michał Karnicki (karni)]

iatll - do you know of
http://developer.android.com/reference/android/content/AbstractThreadedSyncAdapter.html ? Android has a sync framework we could use to avoid too frequent scans, as well as recover from failed scans easily (of which the sync framework would take care of it's own retrying with exponential back-off). Basically, it's a cool thing, but too cool to expand on it here, so if you're not familiar with it, maybe give it a read :) Even if at a later point in time we decide it's not exactly what we need, I think it should be fairly easy to implement sync-up scans in a way which would allow at least trying with SyncAdapter. Basically you implement a sync service, add a few config files, and request syncs when required.

I agree with Axel that "Since the user's really only changing files when the screen is on" is not necessarily true, so we should handle that as well, if we claim it is a _sync_ (up) feature. There are no files we should exclude/ignore by design decision (although by a users choice of other folders, it could be ignored, but it's a different approach).

Re: "And regarding the full scan, we can also allow users to set which folders to scan, decreasing the amount of files we'd have to check." - yes, that is exactly what I've said/wrote :) As I call it, selective sync.

Re: "in my current configuration, wifi is switched off automatically shortly after the screen is turned off, switching back to slower and limited 2g/3g (if enabled)." -- this can be changed by an app/service holding a partial wake lock. However, it is advised to hold wake locks for as short periods of time as possible, to save battery.

Re: "i expected that u1 uses some rsync variant for synchronization so only changed parts of files are actually transferred which makes me wonder why bigger files should not be included in sync" -- I am sorry to inform you it doesn't. If it did, I wouldn't even raise a problem of a large file being modified. It doesn't, and it won't for the foreseeable future, so we should assume full file is re-uploaded.

"u1 client should imho try use as little resources as possible." -- This is naturally the main point of this whole discussion. The question always is how, and this is what we're trying to agree on :)

"The question is - is it ok if the user has set his/her wifi to switch of, say, 2 seconds after screen off and u1 sync service delays this to 4? (or more" - First, we don't know to what value the user has changed wifi switch off. Not to mention the fact that stock roms don't allow you to modify this period (which is a few minutes, I believe, by default). Second, yes it is okey, *if* it meets user expectation, but doesn't impact battery in an unacceptable rate. The golden center between one and another is a delicate matter, which we take very seriously.

Imagine I'm using google navigation for 3 hours, without a charger. GPS is running in background. It's 36 runs of local file scans, if we rescan every 5 minutes. It is a total waste of resources. Now think about this:
* whenever screen is turned on, trigger scan every: 3 minutes, then 6 minutes, then 12 minutes, then every 15 minutes.
  If a file is recognized as modified duri...

[axel (x.)]

crap, just lost my text so i keep this short:

re 'If files are altered externally, would I be correct in assuming that the sdcard would be dismounted first?'
in many cases yes, but sd wont be unmounted if accessed through ftp/ssh/adb for instance.

re ' it takes around 700ms for a scan of 1000 files'
now that sounds a lot better =)

re 'I'm not quite sure, but I think the app currently has you download files individually'
correct, but with the ability to really sync this might no longer be a restriction i hope (see below).

re 'selective sync'
now i guess i get it, sounds great. could selecting a folder 'a' for auto-sync lead to include subfolders 'a1', 'a2', ... for auto sync as well? and what should happen if a subfolder 'a5' is created at a later stage? in my opinion both should be answered with yes since this is more intuitive and closer to the desktop client (u1sd).

re 'increasing intervals depending on scan-results'
good example, good idea! screen off values could go even higher (e.g. 60mins) if you ask me.

re 'We could also employ inexact alarms'
what do you mean by that? and what benefit does this have?

re 'think of what kind of files would users normally change on their phone and hope to get uploaded. In my case it's text files and keepassx files'
ack. plus pictures and videos and maybe small sqlite-databases. btw: i thought the android keepass-client can not alter but only display contents of the keepass database, did you find a solution to also alter keepass-data on your android-device?

[Michał Karnicki (karni)]

Re: downloading whole folders (sync-down), yes.

It should be recursive, yes.

See setInexactRepeating at
http://developer.android.com/reference/android/app/AlarmManager.html
Up side: saves battery.

I don't remember if keepassdroid allows you to modify the db. Basically it's on the "sync down" side for me in case of keepass db file. And the sync down is *much* easier. The sync up is the hard part.

[iatll (iamthelamplighter)]

Michal - I didn't know of AbstractThreadedSyncAdapter, but I do now! Looks perfect for the job.

Ok, I see Michal and Axel's points about files changing while the screen is off. However, we do seem to all agree that seeing the screen turn off is a good indication that having longer (configurable) intervals between scans is OK. The major point of bringing up screen off was to point out that it is OK to not scan every 5 seconds, and that its perfectly fine to increase this into the order of minutes, because we don't have to worry about missing a file modification if we check one more time when the screen goes off.

Increasing the wait time between failed scans is a smart idea, and a good one. However, we must remember that the longer the screen stays on the less relative footprint the app is having. An average scan of 100 files will take about 1/10th of a second to complete! If the screen is on for 3 hours, for example, and 36 scans run, it will be ~3.6 seconds total spent by the CPU over 3 hours (0.03%!). And if another CPU intensive app like GPS is running, the amount of battery taken up by u1 becomes even less significant. Of course, all of this time is still wasted CPU (and network) time, and I think we should definitely make every effort to decrease this, but I think we absolutely should not sacrifice usability & promptness for lite-ness. What if an hour into his GPS-aided drive, the user thinks of a new idea and jots it down on his notepad? He's learned that at the office, the note always syncs immediately after saving, but this time it hasn't because the alarm is set for 40 minutes and now the upload is very, very late. In summary, techniques that decrease the accuracy of the scan are useful for battery life but we need to be very, very careful (and open) about how they work.

I took a look, and yep inexact alarms look best to wake the phone up to do the sync. It is worth noting that although a sync may take 0.1s while the phone is awake, it will likely take at least 10x longer to wake the phone from deep sleep, turn the network on, establish a connection, etc etc etc... this is why I am so much more keen on figuring out exactly how often we need to scan when the screen is off and the phone is asleep!

I'm building the scan service right now, and yes Axel it is recursive and yes it detects new folders as well. A good thing about this is that last modified times work for direct children! So if we have a folder with 300 files in it, and the folder's last modified time has changed...we don't need to read the last modified time of any ofthose files! (but we still iterate through them since they could be directories though)

Axel - I didn't know the sd won't be unmounted if accessed through ftp/ssh/adb. This is important and I will keep it in mind. Thanks!

Michal's suggestion at the bottom of his post is best, I agree with it. Go for simple, start syncing periodically and let the user define an interval he/she is comfortable with (one for screen off, one for screen on). I think the debate above is really more about what the default time should be. And yes, a failed sync will result in some cool-down time and an automatic retry.

[Leonid Evdokimov (darkk)]

My 2 cents.

> A good thing about this is that last modified times work for direct children! So if we have a folder with 300 files in it, and the folder's last modified time has changed...we don't need to read the last modified time of any ofthose files!

It looks like wrong assumption to me. Directory modification means that some files were added/removed/renamed in the directory (and, maybe, something else).

I confirm it on desktop linux using following test-case:
1. mkdir directory
2. cd directory
3. stat .
4. echo "foobar" > file
5. stat .
6. vim file
7. :q (without any editing)
8. stat .

You'll see difference between stat at steps 5 and 8 because of temporary file `vim` creates. `file` is left unchanged.

On the other hand you can do following steps:

6. echo "babababaz" >> file
7. stat .

And you'll see, that `file` was actually changed and directory mtime is same(!).

So, directory modification time does not magically reflect it's children modification times.

[Michał Karnicki (karni)]

I didn't get the "So if we have a folder with 300 files in it, and the folder's last modified time has changed...we don't need to read the last modified time of any ofthose files!" part, you probably meant the folder's last mod time hasn't changed?

Anyway, I do agree with Leonid's observation (thanks!) that you can't base your knowledge on parent dir mod time. I believe it is not very useful from research I did before. Same goes for U1 - I believe if you crate a file, parent dir mod time will change, but not if you update a file (what a pity :( ).

When talking about 0.1s to scan 100 files, there's always some overhead connected to complicated code (that's a very general statement), so benchmarking in practice will probably indicate slightly higher times once we get to test it. I'm just super sensitive about CPU/battery, because we've already had a number of users who uninstalled U1F due to overusing resources (!). Sometimes I blame custom roms, sometimes I blame myself. Anyway, the code is live, and can always get reviewed.

Keep up the great work Derek!

[iatll (iamthelamplighter)]

Oops. Yes, both of you are right, I just tested it again on Android. The directory 's last modification time changes if a file is added, but not if a child file is changed. I got mixed up with the discussion about inotify a couple of posts up. Thanks for correcting me on this, Leonid

Right, there is overhead, and actually it will be a little longer because if we want to have sync-down, we'll have to send an http request off first. The timing is a very crude way of getting an idea of the complexity, because even if it takes 5 seconds it doesn't necessarily mean that it's doing computation the entire time, it could be just in disk i/o or waiting on the network, etc. However, it is a decent benchmark for determining if the method is even worth looking at. I understand about CPU & Battery, and I think that from a user standpoint having control over how often the sync occurs will go a long way towards making the user less concerned about it.

Thanks!

[Robin Sheat (eythian)]

Op 13-09-12 16:26, iatll schreef:
> Right, there is overhead, and actually it will be a little longer
> because if we want to have sync-down, we'll have to send an http request
> off first.

Could you hook into google's notification system thingy to handle this
instead? It would make it a) instant, and b) low resource.

[Michał Karnicki (karni)]

I'm thinking about it Robin, but it's much more complex logistically (and probably not suitable for contributor work, as requiring server side stuff). Also, work by Derek would be worthwhile, because we wouldn't be using GCM any time soon.

[Michał Karnicki (karni)]

(Just to add: GCM would be tied to sync-down, which is the easy part. Sync-up from the local device, detecting changes without wasting resources, is what Derek is looking into in the first place.)

[Michał Karnicki (karni)]

Hey all,
We continue to exchange e-mails with Derek about the sync features. I only started documenting them here:
https://wiki.ubuntu.com/UbuntuOne/Android/UbuntuOneFiles
but feel free to jump in and help out. If you notice any unhandled use cases, feel free to add them to the wiki.
This might be the first serious community contribution to U1F!

[grubernd (grubernd)]

seriously? i thought the basic idea behind Ubuntu One would be to sync files among machines. maybe i was mistaken.

*highly disappointed*

[SB (stefan-bommer)]

Hi, don't know what the recent status update on the auto-syncing is, but I have an idea for you that you might have already considered, but I'm just throwing it out there, because I too would love that feature to come around some time (and I hope that Michael isn't still the only developer on the Android client :) ):

Have you looked at Dropbox' Android client solution? I don't exactly know how they do it technically (because I am not a dev), but they implemented a "favorite" feature where you can star a file or a folder for auto-sync and the client will only check on these specific items for changes. Surely this should reduce the overhead you were talking about in the thread, and it gives the user granular control over which files they want to sync to their phone.

Cheers, keep up the good work!

[Michał Karnicki (karni)]

Hi Stefan,

Thanks for your comment. Indeed, selective sync (think in terms of starring/favoriting/etc) is the only way to go. We'd never make user sync all of their files. Dropbox'es solution is significantly different (also in the fact they only reupload files you edit from within the app, they no longer use regular /sdcard/Dropbox directory to store files).

On an unrelated note, yes, I'm still the only one, it'll take some time before I get to this feature :| I can't wait myself! :)

[iatll (iamthelamplighter)]

Stefan,

I spent some time in the fall last year talking to Michal about the logistics of implementing this. Basically, we found that it's actually very tricky, especially in making a sufficiently efficient (ie battery usage) solution. (Michal's created some very good documentation here https://wiki.ubuntu.com/UbuntuOne/Android/UbuntuOneFiles)

The result of that was, I ended up implementing a working partial solution (which I believe I still have the .apk for if you're interested). All it does is: when you open the app, it scans your u1 folder - if any of the files has been changed, it will upload them. That's it. I've been using it on my phone since the fall to satisfy my uploading needs, haven't had a problem with it so far. However, it's really much more of a hack-on - it hasn't been properly tested or reviewed, at least not by anyone actually involved in the Ubuntu One project, and it really is not complete enough to be called a real solution. =/

[Mike Ashelby (mike-ashelby)]

Hi Guys,

I think this is a definitely required feature. I have a bunch of files that I need access to on my wifi tablet - I don't always have wifi, and I also need the most recent version. Something along the lines of Google Cloud Messaging (http://developer.android.com/google/gcm/index.html) should allow pushing of changes to the device (even giving the size of the necessary download to check if it's too big to download, say on cell network).

The Android syncing back should be taken care of by FileObserver http://developer.android.com/reference/android/os/FileObserver.html, sholdn't it?

I'm not a developer, so sorry if I'm way off track. What do you all think?

Mike