tomboyToHTML converter breaks if unknown entities are in the note - /notes/ OOPS

Bug #539521 reported by Roman Yepishev
176
This bug affects 23 people
Affects Status Importance Assigned to Milestone
Ubuntu One Servers
Fix Released
High
Rodrigo Moya

Bug Description

In case any note contains any non lt/gt/amp entity while it is in XML format, the whole /notes/ web UI OOPSes for the user.

The exceptions of this type are now filling our appaserver stderr logs:

Traceback:
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/django/core/handlers/base.py" in get_response
  86. response = callback(request, *callback_args, **callback_kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/django/contrib/auth/decorators.py" in __call__
  67. return self.view_func(request, *args, **kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/account/djangolib/decorators.py" in wrapper
  30. return func(request, *args, **kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/views.py" in view_notes
  84. note["content"] = tomboyToHTML(note["content"])
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/tomboyxml.py" in tomboyToHTML
  95. return convert(xml, TomboyXMLParser())
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/tomboyxml.py" in convert
  180. for action, elem in context:

Exception Type: XMLSyntaxError at /notes/
Exception Value: Entity 'nbsp' not defined, line 1, column 42

While it is not clear how one can enter   entities via our web editor now, that might have been possible to do in the past.

Therefore there should be some kind of protection against this issue. Specifying the set of known entities to the parser might help. However I also see :
XMLSyntaxError: line 17: Entity 'locale' not defined
XMLSyntaxError: line 1: Entity 'Itemid' not defined
XMLSyntaxError: line 1: Entity 'monitor' not defined
XMLSyntaxError: line 1: Entity 'pageNum' not defined
XMLSyntaxError: line 2: Entity 'search' not defined
XMLSyntaxError: line 2: Namespace prefix fb on fan is not defined
XMLSyntaxError: line 54: Entity 'lang' not defined

It looks like those are the parts of some URL that was pasted to the note and it was not properly saved by the server.

Revision history for this message
Roman Yepishev (rye) wrote :
Changed in ubuntuone-servers:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Ubuntu One Desktop+ team (ubuntuone-desktop+)
tags: added: desktop+ notes webui
Roman Yepishev (rye)
visibility: private → public
Revision history for this message
Roman Yepishev (rye) wrote :

In case entity is not properly formed (i.e. &amp ) then this is the stack trace:

ERROR:root:WSGI Handled Exception (OOPSID 1538appserver40694)
Traceback (most recent call last):
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/django/contrib/auth/decorators.py", line 67, in __call__
    return self.view_func(request, *args, **kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/account/djangolib/decorators.py", line 30, in wrapper
    return func(request, *args, **kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/views.py", line 84, in view_notes
    note["content"] = tomboyToHTML(note["content"])
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/tomboyxml.py", line 95, in tomboyToHTML
    return convert(xml, TomboyXMLParser())
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/tomboyxml.py", line 180, in convert
    for action, elem in context:
  File "iterparse.pxi", line 321, in etree.iterparse.__next__
  File "parser.pxi", line 627, in etree._raiseParseError
XMLSyntaxError: line 1: EntityRef: expecting ';'

Revision history for this message
Rodrigo Moya (rodrigo-moya) wrote :

I guess there's nothing we can do for broken XML, but at least we should not oops. Working on that

Changed in ubuntuone-servers:
assignee: Ubuntu One Desktop+ team (ubuntuone-desktop+) → Rodrigo Moya (rodrigo-moya)
status: Confirmed → In Progress
Roman Yepishev (rye)
Changed in ubuntuone-servers:
importance: Medium → High
Revision history for this message
Roman Yepishev (rye) wrote :

Rodrigo, I believe we need to sanitize input from XML prior to handling that to XML processor, i.e. do whatever we can to escape &'s that are not entities.

Roman Yepishev (rye)
tags: added: current-trouble
Revision history for this message
GonzoDark (gonzo-dark) wrote :

I got this probelm, is anyone working on a fix? :(

and I can't synch my files :S

Revision history for this message
Roman Yepishev (rye) wrote :

The branch Rodrigo proposed will try to parse note contents first and if that fails the note will be rendered as tomboy XML:

E.g. if note contains:
    <monospace>This &is an invalid XML</url></monospace>
Then it will be displayed as is on the web page.

While it is not possible to recover note formatting automatically this change will grant the access to the contents of the note,

Changed in ubuntuone-servers:
status: In Progress → Fix Committed
Revision history for this message
Eemil Lagerspetz (eemil-lagerspetz) wrote :

My note has Chinese in both the title and the content, and I am unable to edit/delete the note. I saw a terminal tool before to do this, but now https://couchdb.one.ubuntu.com/ says service unavailable and the console tool fails to work. Any suggestions?

何洋 (he-yang)
Changed in ubuntuone-servers:
status: Fix Committed → Fix Released
status: Fix Released → In Progress
Changed in ubuntuone-servers:
status: In Progress → Fix Released
kennethdurr (kendu0301)
Changed in ubuntuone-servers:
assignee: Rodrigo Moya (rodrigo-moya) → kennethdurr (kendu0301)
Revision history for this message
Roman Yepishev (rye) wrote :

Please do not change the owner unless you plan to fix the issue. And this issue has been fixed for quite a while. Thank you.

Changed in ubuntuone-servers:
assignee: kennethdurr (kendu0301) → Rodrigo Moya (rodrigo-moya)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.