Creole parser assumes ASCII

Bug #614286 reported by Jeroen T. Vermeulen on 2010-08-06
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Wikkid Wiki
Undecided
Jeroen T. Vermeulen

Bug Description

The creole parser breaks on non-ASCII characters.

Traceback (most recent call last):
  File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/user/bzr.py", line 40, in __call__
    resp = req.get_response(self.app)
  File "/usr/lib/pymodules/python2.6/webob/request.py", line 918, in get_response
    application, catch_exc_info=False)
  File "/usr/lib/pymodules/python2.6/webob/request.py", line 886, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/app.py", line 77, in __call__
    response = view.render(self.skin)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/view/base.py", line 121, in render
    return self._render(skin)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/view/wiki.py", line 42, in _render
    return super(WikiPage, self)._render(skin)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/view/base.py", line 111, in _render
    content = template.render(**self.template_args())
  File "/usr/lib/pymodules/python2.6/jinja2/environment.py", line 705, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/skin/default/page.html", line 1, in top-level template code
    {% extends "base.html" %}
  File "/usr/local/lib/python2.6/dist-packages/wikkid/skin/default/base.html", line 43, in top-level template code
    {% block content %}{% endblock %}
  File "/usr/local/lib/python2.6/dist-packages/wikkid/skin/default/page.html", line 3, in block "content"
    {% block content %}{{ view.content }}
  File "/usr/lib/pymodules/python2.6/jinja2/environment.py", line 352, in getattr
    return getattr(obj, attribute)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/view/wiki.py", line 31, in content
    return formatter.format(self.context.base_name, content)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/formatter/creoleformatter.py", line 33, in format
    return HtmlEmitter(Parser(text, self.rules).parse()).emit()
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 410, in parse
    self.parse_block(self.raw)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 405, in parse_block
    re.sub(self.rules.block_re, self._replace, raw)
  File "/usr/lib/python2.6/re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 394, in _replace
    replace(groups)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 305, in _text_repl
    self.parse_inline(text)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 400, in parse_inline
    re.sub(self.rules.inline_re, self._replace, raw)
  File "/usr/lib/python2.6/re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 394, in _replace
    replace(groups)
  File "/usr/local/lib/python2.6/dist-packages/wikkid/contrib/creole_1_1/creole.py", line 385, in _char_repl
    self.text.content += groups.get('char', u'')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Related branches

Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Attaching a branch that seems to make it work for me, on cursory inspection. It's probably all wrong etc. so not bothering to write tests. Just an illustration.

Revision history for this message
Tim Penhey (thumper) wrote :

I'll take something that appears to work over nothing at all. We can write more tests as we go :)

Changed in wikkid:
status: New → Fix Committed
milestone: none → 0.2
assignee: nobody → Jeroen T. Vermeulen (jtv)
status: Fix Committed → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers