Two-way synchronization

[Benoit Garret]

The work I've done on the web-sync branch only works in the case of a one-way synchronization. Olivier is working on note editing, which will break the current synchronization code.

I'd rather plan this ahead than have to remove the sync feature. The current Tomboy code can be used as an example, it shouldn't be difficult to implement but some changes to the database schema will be needed.

[Rodja (trappe)]

Sandy Armstrong wrote on 2010/03/24 to the mailing list:

---------8<-----
The overall two-way sync algorithm is in this giant method:

http://git.gnome.org/browse/tomboy/tree/Tomboy/Synchronization/SyncManager.cs#n362

The code that is specific to syncing with the Tomboy Web REST API is
in the WebSyncService add-in, and perhaps the best place to start
looking is here:

http://git.gnome.org/browse/tomboy/tree/Tomboy/Addins/WebSyncService/WebSyncServer.cs
--------->8-----

[Rodja (trappe)]

I started developent in lp:~tomdroid-dev/tomdroid/two-way-sync which is branched from lp:~tomdroid-dev/tomdroid/sync-ui and owned by the tomdroid-dev team.

Currently I try to understand how the two-way syncing is done in Tomboy and do some minor refactorings within org.tomdroid.sync.web to clean the path.

[Sandy Armstrong (sanfordarmstrong)]

Please let me know if you have any questions about Tomboy sync. Recent messages on tomboy-list from the Conboy developers on this topic might also be of interest.

[Rodja (trappe)]

I've done some refactoring and started to implement unit tests for the two way synchronisation. Right now it "should" again do one-way-synchronization. I can't test it at the moment because bug https://bugs.launchpad.net/ubuntuone-servers/+bug/577192 has hit me and the ubuntu one people need to fix it before I can go on with real live tests. I guess you will notice a big speed up in sync, because now only the diff up to the last synced version are fetched from the server.

[Rodja (trappe)]

The U1 bug is fixed and I've succsessfully synced all my notes by using the newly havy refactored code. The next step will be to implement uploading changed Notes to the remote server and hence implement basic editing of notes.

Until now I've come up with three questions regarding the Tomboy sync code:

1. Aquiring Lock
Is a lock aquired by connecting to the remote server and making shure we get a current-sync-guid as it seems to be done in Tomboy (WebSyncServer.cs:62)? How is the unlock done?

2. Resetting Client
In SyncManager.cs:425 happens check to make sure the current-sync-guid is the same as the one known by the client. Otherwise the client is resetted. The comment indicates this is needed to make sure local notes are not deleted when the server is wiped/reinitialized. Will the server change it's guid when wiped? Also I wonder why a reset is going to help in such a situation.

3. Title Conflicts
What is the purpose of the title conflict resolving code (SyncManager.cs:450)? It seems to search for notes which
 * have changed/created on the server since last update
 * are not found via UUID on the local side, but
 * have the same title as a local Note but different content
If I unserstand correctly, this can only happen when two clients seperately create a note with same title. Must the title be unique within Tomboy? Otherwise I do not understand why this title conflict code is needed at all...

[Sandy Armstrong (sanfordarmstrong)]

On Mon, Jun 21, 2010 at 3:37 AM, Rodja <email address hidden> wrote:
> The U1 bug is fixed and I've succsessfully synced all my notes by using
> the newly havy refactored code. The next step will be to implement
> uploading changed Notes to the remote server and hence implement basic
> editing of notes.
>
> Until now I've come up with three questions regarding the Tomboy sync
> code:
>
> 1. Aquiring Lock
> Is a lock aquired by connecting to the remote server and making shure we get a current-sync-guid as it seems to be done in Tomboy (WebSyncServer.cs:62)? How is the unlock done?

There is no actual lock acquisition in web sync. Uploading notes is
done as a single request in which you include the new sync revision
you expect you'll be creating. If somebody else has done a sync by
the time your request hits the server, the server will say "uh oh,
you'd actually be creating rev N+1, not N, so I'm not going to accept
this sync", and there will be an error PUTing the request.

Lock acquisition is only needed in the case where we're faking a
server, like with the fuse-based sync backends.

> 2. Resetting Client
> In SyncManager.cs:425 happens check to make sure the current-sync-guid is the same as the one known by the client. Otherwise the client is resetted. The comment indicates this is needed to make sure local notes are not deleted when the server is wiped/reinitialized. Will the server change it's guid when wiped? Also I wonder why a reset is going to help in such a situation.

A properly implemented websync server would change its per-user guid
if the user decided to wipe his notes and start over at rev 0. If a
user was at rev 5 (for example), then wiped his notes and went back to
0, and sync'd 5 times with Tomboy before ever syncing again with
Tomdroid, and we did not have the per-user guid, Tomdroid would think
the server was in the same state as the last time it sync'd based on
rev alone.

This assumption would be wrong and could easily lead to data loss.

> 3. Title Conflicts
> What is the purpose of the title conflict resolving code (SyncManager.cs:450)? It seems to search for notes which
>  * have changed/created on the server since last update
>  * are not found via UUID on the local side, but
>  * have the same title as a local Note but different content
> If I unserstand correctly, this can only happen when two clients seperately create a note with same title. Must the title be unique within Tomboy? Otherwise I do not understand why this title conflict code is needed at all...

Titles need to be unique in Tomboy.

Hope this helps!

[Rodja (trappe)]

Finally I've got some time to drive the two-way-sync branch a little further down the road: I sucsessfully synced forth and back some notes between Tomboy and Tomdroid using the Ubuntu One webservice. Right now I recommend to avoid collision scenarios. But I would be glad if a few people could give it try and review the code I've written.

Remaining tasks:

  * deleting and creating of notes
  * title merging to ensure uniquness
  * ensure connection errors will not leave client in an undefined state
  * merging with the newer code in the sync-ui branch

Anyone interested in helping me to push this further? I've got to less spare time to get this working as fast as I would like.

[Guilherme Salgado (salgado)]

Hi Rodja,

I'd be really happy to help pushing this forward, but I think it's
really important that we try to get the one-way web sync branch merged
onto trunk before we do so. I think that's important because there
already are a bunch of non-trivial conflicts when merging the
two-way-sync branch into lp:~tomdroid-dev/tomdroid/sync-ui.

Since the sync-ui doesn't seem to have any critical bugs or regressions,
I think it'd be worth to merge that into trunk, merge trunk into the
two-way-sync branch and then continue with the remaining tasks that you
described. How does that sound to you?

If you like that plan, how would you feel about proposing
lp:~tomdroid-dev/tomdroid/sync-ui for merging into trunk? I'd be happy
to try and get Olivier to review it, as well as doing any changes he
finds necessary.

[Rodja (trappe)]

Hi Guilherme,

I strongly support your suggestion to first merge sync-ui into master before putting more work into two-way-sync. I would not have started this befor feeling good about the sync-ui. I tought everthing was on it's way for a 0.4 release. Now I double back and just did the merge proposal as you asked me. Hopefully we will see a new release soon.

[Rodja (trappe)]

The code in lp:~tomdroid-dev/tomdroid/two-way-sync aiming to fix Bug #549647 shows the raw XML text in Note Editing Mode. To prevent input errors, gain better overview and simplify editing, the Editing Mode should render very similar to the View Mode (except the clickable Hyperlinks).

[Rodja (trappe)]

Oh, the last message should not have been posted here but as a description of Bug #657568.

I've started to create some new bugs for tasks not directly related to the core functionality of two-way-sync.

[Rodja (trappe)]

I've used two-way-sync for a while without problems so I've proposed merging into master for the next release. We could then start tackling the remaining usability issues in seperated branches without loosing focus.

[Kieran Hogg (xerosis)]

Has there been any progress on this?

[Olivier Bilodeau (plaxx)]

Benoît or myself need to review the branch changes, (maybe) perform fixes and write down remaining clean-up to be done later.

I have not started to look into it as my life has been pretty rock and roll since the newborn..

[Stefan Hammer (j-4-deactivatedaccount)]

Is there anything we can do to help you releasing the two way sync? Or at least a experimental version for testing?

[Olivier Bilodeau (plaxx)]

Checkout the branch, run it and let us know if any problems. Optionally you
can try to update it from trunk if you are familiar with this process.

[Olivier Bilodeau (plaxx)]

Closing this. Please use latest beta series release if you want two-way sync. Any problem found should be filed separately.