Sync Tomboy notes lose bullet-list structure. >:-(

[Brewster Malevich]

STEPS TO REPRODUCE/TEST:

Two Karmic computers with all the latest system updates. Both have Tomboy setup to synchornize to the same Ubuntu One account.

Computer 1
1. Copy http://launchpadlibrarian.net/39307693/cf2a19b9-a8f9-42cd-87f1-8642c69fa9cd.note to ~/.local/share/tomboy/
2. Open Applications->Accessories->Tomboy
 2a. Make sure "Bullet List Test" is there & looks like http://launchpadlibrarian.net/39308179/tomboy_bullet_list_test_screenshot.png
3. Tools->Synchronize Notes

Computer 2
4. Open Applications->Accessories->Tomboy
5. Tools->Synchronize Notes

Result: Note's bullets are lost - http://launchpadlibrarian.net/39308774/tomboy_bullet_list_test_bad_screenshot.png

Expected Result: Note is the same as on Computer 1 - http://launchpadlibrarian.net/39308179/tomboy_bullet_list_test_screenshot.png

-------------

ORIGINAL DESCRIPTION:

Binary package hint: ubuntuone-client

This may also be an issue with U1 server, or Tomboy.

I have one hundred or so tomboy notes that I've synced to U1. Several of these notes are paragraphs structured as bulleted lists with many levels. However, after I've synced these with U1, all of my bullets are shifted-left to the highest level. This is -very- un-cool.

ProblemType: Bug
.home.sbm..cache.ubuntuone.log.syncdaemon.exceptions.log:

.home.sbm..config.ubuntuone.syncdaemon.conf:
 [bandwidth_throttling]
 on = True
 read_limit = 102400
 write_limit = 102400
.home.sbm..config.ubuntuone.ubuntuone.client.conf:
 [ubuntuone]
 bookmarked = True
 connected = True
 connect = 0
 show_applet = 1
Architecture: amd64
Date: Fri Nov 13 15:35:08 2009
DistroRelease: Ubuntu 9.10
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
NonfreeKernelModules: wl
Package: ubuntuone-client 1.0.2-0ubuntu2
PackageArchitecture: all
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.48-generic
SourcePackage: ubuntuone-client
Uname: Linux 2.6.31-14-generic x86_64

[Brewster Malevich (brews)]

Here is a screenshot for a note I had just made showing the formatting in tomboy (right window), and the formatting in the web UI (left window). So, I'm guessing that they are cleaned of their original format when the note is downloaded from U1, if you have the original note, though, the format seems to remain the same.

I'm changing this over to U1 server.

[Henning Eggers (henninge)]

This bug affects me, too, and it looks even worse than the screen shot. See may attached screen shots.

It seems that it happens upon de-indention. Looking at source code of the first web version of the note, I see <li> tags outside an <ul>. Does not look to hard to fix ... ;-)

[Sandy Armstrong (sanfordarmstrong)]

Did you ever edit the note in U1's web interface?

[Rodrigo Moya (rodrigo-moya)]

This is a bug on the XML<->HTML conversion code we have in U1 server. I'm working on a new converter, which should fix it and should be landing soon

[Sandy Armstrong (sanfordarmstrong)]

Just ran across this bug, even though I never edited the note on U1.

This seems to imply that you guys are storing the note in HTML on the server, and converting back and forth during sync. Caching the HTML version is probably a good idea, but still it seems safer that when I send a note update via JSON, U1 would store the exact values I send instead of converting back and forth.

I'm working on a Tomboy sync feature that reduces visible conflicts by comparing note content in the case that modification times imply a conflict. But testing this feature with U1 is not working very well because so many of my notes are mangled. And this feature would work a lot better if the content synchronized to U1 was only modified when the user uses your editor (Otherwise, even if your conversion bugs are worked out, there are bound to be little differences that Tomboy will interpret as conflits).

[Sandy Armstrong (sanfordarmstrong)]

Two things I forgot:

1) Breaking users' bulleted lists is a really big deal; the importance of this bug should be raised, since people could easily lose data without realizing it.

2) I'd like to file a new bug against the U1 servers about how note content is stored, but would like some confirmation that my guess about back-and-forth conversion is correct.

[Brewster Malevich (brews)]

I have lost data and moved to Dropbox because of this. I was so angry (as you can see by the angry face in the report title). I was slowly easing into U1. Tomboy notes and a few loose .pdfs were my guinea pig. The biggest casualty were zoology notes which are ~very~ long and quite dependent on hierarchical structure in bulleted lists. They are now barely-coherent mush. I'm not going to rant, though I would like to.

[Sandy Armstrong (sanfordarmstrong)]

Is there any progress on this? This is a pretty big data loss bug.

If there is not a fix soon, I'm going to have to start warning Tomboy users away from U1 until it's fixed. I've been avoiding mentioning this bug because I assumed a fix was imminent and I didn't want to cast a negative light on the excellent work you guys have been doing. But at this point I feel it is unfair of me to let users think U1 is perfectly "safe" for their notes.

If there's anything I can do to help, please let me know.

[Rodrigo Moya (rodrigo-moya)]

I have been working on a new HTML<->XML converter, to fix this and other bugs we have opened, but due to some limitations in the XML parser we use, I haven't been able to fix all the bugs in the same branch, so I guess I should try to fix this one separately, without waiting for the new HTML<->XML converter

Sorry for the delay, will work on it tomorrow

[Henning Eggers (henninge)]

I very strongly agree with Sandy about not converting notes back-and-forth when storing to and retrieving from U1. The converter should only be used when displaying and editing the notes through the web UI. This is especially important as you are implying that there are more issues with the converter that have not been fixed yet.

Can you please confirm that the behavior on the server is as Sandy guessed? Or has that bug already been filed against the servers?

[Sandy Armstrong (sanfordarmstrong)]

Rodrigo, Stuart, and I discussed this bug this morning, along with the underlying issue of U1 converting all notes to HTML and storing them that way.

I think we're all agreed that the best approach is to store the native XML format, and only convert to HTML when viewing/editing via U1.

My understanding is that once the new converter is ready, everyone's notes in U1 will be converted back from HTML into native XML, and that will be the format used for sync; no more converting will be done (except, obviously, if you use the U1 note editor).

I'm going to help by providing a few good examples of complex note XML.

[Jeff Bauer (jbauer)]

I attached an example note in my bug report #499823, which appears to be a dupe of this bug, #482372. I've attached it here now in case my earlier report is closed. I'm glad to see the status raised to critical, as U1 is unusable for syncing tomboy notes in its current state.

[Konrad Voelkel (konradvoelkel)]

Okay, now I understand why I had these HUGE issues about the syncing. I had quite a big data loss because of this. I suggest it would be very polite to put a big warning in Tomboys sync config menu as well as on the Ubuntu One page as long as this bug remains. This is something that will move people away from Tomboy!

[Sandy Armstrong (sanfordarmstrong)]

Konrad, this is not a Tomboy bug, it is an Ubuntu One bug. Adding a warning in Tomboy wouldn't make any sense because it would only apply to people using web sync with U1, and Rodrigo will be fixing it soon anyway, so the warning would be obsolete pretty quickly.

I agree it's a huge issue and am concerned myself that there is no sort of warning at one.ubuntu.com about this data loss bug.

[Cornelius Hald (hald)]

Sorry for my ignorance, but does "Fix Committed" mean that the bug is fixed on the production system, or will the fix still have to be released, packaged, deployed, etc.?

[Henning Eggers (henninge)]

Yes, it means it's in the source tree but has not been deployed yet. "Fix Released" is what you are waiting for.

[Rodrigo Moya (rodrigo-moya)]

Yes, it means that the code is in a bzr branch, waiting to be approved, and then it will get deployed, but that takes a bit of time, sorry.

Hopefully it will get deployed to a test server soon, so you can test it. I'll try to remember to send you a mail, Cornelius, once it's deployed to the test server

[Joshua Hoover (joshuahoover)]

A note for reproducing bug and testing fix.

[Joshua Hoover (joshuahoover)]

Screenshot of http://launchpadlibrarian.net/39307693/cf2a19b9-a8f9-42cd-87f1-8642c69fa9cd.note

[Joshua Hoover (joshuahoover)]

Screenshot of http://launchpadlibrarian.net/39307693/cf2a19b9-a8f9-42cd-87f1-8642c69fa9cd.note

[Joshua Hoover (joshuahoover)]

Screenshot of http://launchpadlibrarian.net/39307693/cf2a19b9-a8f9-42cd-87f1-8642c69fa9cd.note after sync to 2nd computer. Bullets are lost.

[Carlos Romel (carlos-romel)]

Notes with bullet marks can't sync sucessfully on web interface.

[Carlos Romel (carlos-romel)]

Notes with bullet marks can't sync sucessfully on web interface. (result on tomboy).

[Rodrigo Moya (rodrigo-moya)]

Carlos, yes, this is known, the web editor is a bit broken in regards to nested lists. We are looking at replacing it with another editor, that generates well-formed HTML. Now at least you shouldn't lose the nested lists structure if you just edit notes on Tomboy/Conboy

[Rodrigo Moya (rodrigo-moya)]

BTW, add further comments about this to bug #524555