tomboyToHTML converter breaks if unknown entities are in the note - /notes/ OOPS

[Roman Yepishev]

In case any note contains any non lt/gt/amp entity while it is in XML format, the whole /notes/ web UI OOPSes for the user.

The exceptions of this type are now filling our appaserver stderr logs:

Traceback:
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/django/core/handlers/base.py" in get_response
  86. response = callback(request, *callback_args, **callback_kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/django/contrib/auth/decorators.py" in __call__
  67. return self.view_func(request, *args, **kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/account/djangolib/decorators.py" in wrapper
  30. return func(request, *args, **kwargs)
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/views.py" in view_notes
  84. note["content"] = tomboyToHTML(note["content"])
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/tomboyxml.py" in tomboyToHTML
  95. return convert(xml, TomboyXMLParser())
File "/home/rtg/canonical/ubuntuone/trunk-2a/utilities/../lib/ubuntuone/web/notes/tomboyxml.py" in convert
  180. for action, elem in context:

Exception Type: XMLSyntaxError at /notes/
Exception Value: Entity 'nbsp' not defined, line 1, column 42

While it is not clear how one can enter   entities via our web editor now, that might have been possible to do in the past.

Therefore there should be some kind of protection against this issue. Specifying the set of known entities to the parser might help. However I also see :
XMLSyntaxError: line 17: Entity 'locale' not defined
XMLSyntaxError: line 1: Entity 'Itemid' not defined
XMLSyntaxError: line 1: Entity 'monitor' not defined
XMLSyntaxError: line 1: Entity 'pageNum' not defined
XMLSyntaxError: line 2: Entity 'search' not defined
XMLSyntaxError: line 2: Namespace prefix fb on fan is not defined
XMLSyntaxError: line 54: Entity 'lang' not defined

It looks like those are the parts of some URL that was pasted to the note and it was not properly saved by the server.

[Roman Yepishev (rye)]

We need to deal with this after bug #527374 - see https://bugs.launchpad.net/ubuntuone-servers/+bug/527374/comments/1

[Roman Yepishev (rye)]

In case entity is not properly formed (i.e. &amp ) then this is the stack trace:

ERROR:root:WSGI Handled Exception (OOPSID 1538appserver40694)
Traceback (most recent call last):
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/django/contrib/auth/decorators.py", line 67, in __call__
    return self.view_func(request, *args, **kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/account/djangolib/decorators.py", line 30, in wrapper
    return func(request, *args, **kwargs)
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/views.py", line 84, in view_notes
    note["content"] = tomboyToHTML(note["content"])
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/tomboyxml.py", line 95, in tomboyToHTML
    return convert(xml, TomboyXMLParser())
  File "/srv/ubuntuone.com/production/ubunet-rev-2450/utilities/../lib/ubuntuone/web/notes/tomboyxml.py", line 180, in convert
    for action, elem in context:
  File "iterparse.pxi", line 321, in etree.iterparse.__next__
  File "parser.pxi", line 627, in etree._raiseParseError
XMLSyntaxError: line 1: EntityRef: expecting ';'

[Rodrigo Moya (rodrigo-moya)]

I guess there's nothing we can do for broken XML, but at least we should not oops. Working on that

[Roman Yepishev (rye)]

Rodrigo, I believe we need to sanitize input from XML prior to handling that to XML processor, i.e. do whatever we can to escape &'s that are not entities.

[GonzoDark (gonzo-dark)]

I got this probelm, is anyone working on a fix? :(

and I can't synch my files :S

[Roman Yepishev (rye)]

The branch Rodrigo proposed will try to parse note contents first and if that fails the note will be rendered as tomboy XML:

E.g. if note contains:
    <monospace>This &is an invalid XML</url></monospace>
Then it will be displayed as is on the web page.

While it is not possible to recover note formatting automatically this change will grant the access to the contents of the note,

[Eemil Lagerspetz (eemil-lagerspetz)]

My note has Chinese in both the title and the content, and I am unable to edit/delete the note. I saw a terminal tool before to do this, but now https://couchdb.one.ubuntu.com/ says service unavailable and the console tool fails to work. Any suggestions?

[Roman Yepishev (rye)]

Please do not change the owner unless you plan to fix the issue. And this issue has been fixed for quite a while. Thank you.