Impossible to remove a corupted note from the server

[sinarf]

I cannot sync for a couple of week.

there is the relevant part of the debug log :
------------------
[DEBUG]: Building web request for URL: https://one.ubuntu.com/notes/api/1.0/op/?include_notes=true
[ERROR]: Synchronization failed with the following exception: 'note-content' is expected Line 6, position 4.
  at Mono.Xml2.XmlTextReader.Expect (System.String expected) [0x00000]
  at Mono.Xml2.XmlTextReader.ReadEndTag () [0x00000]
  at Mono.Xml2.XmlTextReader.ReadContent () [0x00000]
  at Mono.Xml2.XmlTextReader.Read () [0x00000]
  at System.Xml.XmlTextReader.Read () [0x00000]
  at Tomboy.Sync.NoteUpdate..ctor (System.String xmlContent, System.String title, System.String uuid, Int32 latestRevision) [0x00000]
  at Tomboy.WebSync.WebSyncServer.GetNoteUpdatesSince (Int32 revision) [0x00000]
  at Tomboy.Sync.SyncManager.SynchronizationThread () [0x00000]
---------

I did a litle of troubleshooting and I found that one of my note cannot be displayed with the Web Interface. The title of the faulty note is "Qualité défauts", the problem will probably be solved by deleting the note from the server but I cannot access the delete button because of the server crash. The note has been removed on my side with no effect.

It would be nice that to be able to see the xml on the web server or at least offering the possibily to delete a note without displaying it.

Please contact me if I can be of any help.

Keep on the good work. :D

[sinarf (sinarf)]

I cancelled my subscription to ubuntu one, I'll will reactivate it in a few days hoping the notes will be deleted then.

[sinarf (sinarf)]

Subcription redone today but the notes are still there.

Please, remove the corrupted note on the server, I tried everything on my side or at least tell that is not possible or what I could do to solve the problem.

thanks

[Roman Yepishev (rye)]

Hello,

In order to remove the note you will need to use ubuntuone-couchdb-query script available from
http://people.canonical.com/~roman.yepishev/ubuntuone-scripts/ubuntuone-couchdb-query

You will need to find out the id of the note that is causing the error. When you click the title for corrupted note you will get the URL in your address bar similar to https://one.ubuntu.com/notes/edit/87a788ca-4c7f-4212-9b1e-6e9560f18e96

The part after /edit/ - 87a788ca-4c7f-4212-9b1e-6e9560f18e96 is your note identifier.

Now you are able to fetch the note from the server (please replace the $id with the identifier found in previous step):

$ python ubuntuone-couchdb-query notes/$id

This will print out the note from the server. It will be in internal Tomboy format but it will be readable. You might want to save the note in case it contains important information. In case this note does not contain any sensitive information, could you please attach the saved copy to this bug report in order to find out the reason of such corruption?

The output of the command above will include URI line as in:
URI: 87a788ca-4c7f-4212-9b1e-6e9560f18e96?rev=1-4993784180b7046d64d161f969e64083

In order to remove the note use the following statement (please replace $uri with the URI found in previous step):

$ python ubuntuone-couchdb-query --http-method=DELETE notes/$uri

This should return something like:
{'rev': '2-4993784180b7046d64d161f969e64083', 'ok': True, 'id': '$id'}

If there is 'ok': True then removal completed successfully.

I understand that this is a lot of manual operations but since this is not expected to happen on a regular basis there is no user-friendly interface for such removals.

[sinarf (sinarf)]

Hello,

Thanks for the response.

I tried the script you linked but there is the result :
$ python ubuntuone-couchdb-query notes/68f1b47f-6e60-48b8-aab7-f3fd0de5556f
Traceback (most recent call last):
  File "ubuntuone-couchdb-query", line 147, in <module>
    show_tokens=options.show_tokens)
  File "ubuntuone-couchdb-query", line 80, in request
    raise ValueError("Error retrieving user data (%s)" % (resp['status'], url))
NameError: global name 'url' is not defined

It's been a while since I played with Python, I gave it a look but I found nothing obvious. Any idea of what I did wrong? (the file has been downloaded using wget. I haded the script to the bug so you can see if there is any problem with it

Note : I don't mind the manual process, because I might learn something during it. ;)

[sinarf (sinarf)]

Hi,

It seems that the cancellation of the subscription has kick out. All my data have been deleted from the server, which is fine for me because it solve my problem and I didn't loose any notes.

Sorry I couldn't retrieve the faulty notes before if was erased, keep on the good work, guys!

This bug can be mark as resolved.

[sinarf (sinarf)]

Today the broken appeared again. I'm gonna try to debug the script to edit the note and send it to you.

[sinarf (sinarf)]

I did an other try on the script and now I have an other error:
$ python ubuntuone-couchdb-query notes/68f1b47f-6e60-48b8-aab7-f3fd0de5556f
Traceback (most recent call last):
  File "ubuntuone-couchdb-query", line 148, in <module>
    show_tokens=options.show_tokens)
  File "ubuntuone-couchdb-query", line 49, in request
    raise Exception("Unable to retrieve your Ubuntu One access details "
Exception: Unable to retrieve your Ubuntu One access details from the keyring. Try connecting your machine to Ubuntu One.

[papukaija (papukaija)]

This bug is affecting 3 people. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. Thank you again for taking the time to report this bug and helping to make Ubuntu One better. Please submit any future bugs you may find.

[sinarf (sinarf)]

There is a number I get when i try to display the faulty note on the server: OOPS-ID-1609appserver78007

What can I do to delete the note on the server? The script provided doesn't seem to work.

[Cornelius Hald (hald)]

I'm also trying to delete corrupted notes from the server, but the script tells me that the "Service is Temporarily Unavailable". It that message true or is it a problem with the script? Accessing non-corrupted notes via sync and web interface is working fine.

conny@XXX:~$ ./ubuntuone-couchdb-query notes/485387bc-xxxx-yyyy-zzzz-e5fa383268f1
There was a problem processing the request:
status:503, response: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>503 Service Temporarily Unavailable</title>\n</head><body>\n<h1>Service Temporarily Unavailable</h1>\n<p>The server is temporarily unable to service your\nrequest due to maintenance downtime or capacity\nproblems. Please try again later.</p>\n<hr>\n<address>Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.8 OpenSSL/0.9.8g Server at couchdb.one.ubuntu.com Port 443</address>\n</body></html>\n'

[sinarf (sinarf)]

The script has been updated and now I have the exact same message.

I have give up on the idea of using ubuntu one. With this kind of service you can accept it to be down for a few hours maybe a day but a few weeks? I make it plain useless.

I very frustrated because I had solve my problem by erasing my account for a few days but the corrupted data came.

I won't even go on the subject that when you unsubscribe it is said that the data will be deleted and even after 2 weeks it is not which is a big problem by itself.

[Tomas 'tt' Krag (tt)]

I think i'm seeing the same bug although i have no idea how to find out which note might be corrupted. I've attached debug logs from tomboy FYI.

I've tried every combination of deauthing and reauthing with ubuntu one, as well as tried edge.one.ubuntu.com but so far with the exact same result.

[sinarf (sinarf)]

Finding the corrupted note shouldn't be too hard. On my case, I just have to try to open notes with the web interface, if the server cashes the note is corrupted.

I really don't what is so hard on the server side to prevent this crashed and allowing to deleted the note.

[Tomas 'tt' Krag (tt)]

sinarf, I get that, but I have 700 notes, and it would take me a while to click on each note independently. Hence my comments above. Should have been more clear in my description.

[sinarf (sinarf)]

You are right it probably doesn't worth the trouble because the fact is that once you've found the note there still no solution. I have found what note causes the problem 2 month ago and there is still no solution. That why I gave up on the all thing.

[Rodrigo Moya (rodrigo-moya)]

we shouldn't be crashing anymore on corrupted notes, AFAIK, but just don't do any XML<->HTML conversion for them at all, so once you find the corrupted note, you should be able to edit it in the web UI or Tomboy (and sync from it), so please do that if possible

[Cornelius Hald (hald)]

I can confirm this. At least I was just able to deleted two corrupted notes using the web frontend.

[sinarf (sinarf)]

Ok, I just tried to delete the note from the web interface and it worked.

I changed the status of this bug since my problem is solve. I'm gonna switch back to ubuntu one for syncing and give it another chance, just because I like the idea.

Thanks.

[Tomas 'tt' Krag (tt)]

That still leaves those of us with 700+ notes a mammoth task of finding out which notes are corrupted. In other words, this isn't a useful solution for me.

What is the right process in this case, should i open a new bug with similar content, or reopen this one?

[sinarf (sinarf)]

My problem described in this bug report has been solve as I could delete the note from the server, and now I can sync again.
For your problem, I not a ubuntu developer and I don't know what the rule is.
But, I see a bit of difference between our two problems, I knew from day one which note was corrupted and I needed to be able to delete it. My problem is clearly solved, that's why I change the status of the bug.

Your problem is more "how to find which note is corrupt on the server ?". My guess is that you should open a new bug with this question as the title. But I have no better solution than the manual solution I used. In my opinion, reopen this bug will just add more confusion. I changed the title of this bug so it reflect better what it does. The original title was way to general (my bad).

I hope you'll find a solution sooner than I found mine.

[Rodrigo Moya (rodrigo-moya)]

I guess the best thing would be to find the faulty note. I know it's tedious, but the problem was with HTML generated from the web editor that was completely wrong. This is now fixed, and we don't crash anymore on wrong HTML/XML, but there's nothing we can really do about your corrupted note.

Some tips though:
* the note was probably saved with the web editor before the fixes, so you probably haven't edited it for a long time
* now for broken HTML/XML, we just escape all tags and display that, so if you haven't seen weird notes' contents on either Tomboy or the web UI, it's probably a note you don't use too often. So, I would just suggest you leave it as it is, and when you see it, you just fix it editing it on the web UI or Tomboy.

Of course, as I said, we shouldn't be crashing on the web UI, so if you see that, please reopen the bug

[Tomas 'tt' Krag (tt)]

OK, After 2½ hours of clicking and scrolling I finally found and fixed the faulty note, and managed to sync for the first time in a month or so. Back in business. cheers

[Rodrigo Moya (rodrigo-moya)]

Glad to hear that Tomas, and sorry for making you do the tedious work