str type used in composer.py - causing UnicodeDecodeErrors

Bug #304456 reported by Russ Ferriday
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Singing & Dancing
New
Undecided
Daniel Nouri

Bug Description

trunk -r HEAD

I trapped the exception with some diagnostic to prove that its the header_text that's problematic. At this point it is a <str> and should be unicode. It's failing on EN DASH == \xe2\x80\x93

This may relate to the other UDE bugs around.

2008-12-02 16:20:22 ERROR collective.singing name : header_text exception : 'ascii' codec can't decode byte 0xe2 in position 1876: ordinal not in range(128) text <string.Template object at 0x9fb7f90>
Traceback (most recent call last):
  File "/opt/zope/stashes/eggs/collective.dancing-0.8.2-py2.4.egg/collective/dancing/composer.py", line 177, in _vars
    subs(name)
  File "/opt/zope/stashes/eggs/collective.dancing-0.8.2-py2.4.egg/collective/dancing/composer.py", line 174, in subs
    vars[name] = string.Template(vars[name]).safe_substitute(vars)
  File "/opt/zope/python244/lib/python2.4/string.py", line 205, in safe_substitute
    return self.pattern.sub(convert, self.template)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1876: ordinal not in range(128)

The traceback is here...

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module Products.PDBDebugMode.pdbzope.runcall, line 60, in pdb_runcall
  Module ZPublisher.Publish, line 42, in call_object
  Module collective.dancing.browser.preview, line 62, in __call__
  Module collective.singing.scheduler, line 99, in render_message
  Module collective.dancing.composer, line 228, in render
  Module collective.dancing.composer, line 177, in _vars
  Module collective.dancing.composer, line 174, in subs
  Module string, line 205, in safe_substitute
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1876: ordinal not in range(128)
> /opt/zope/python244/lib/python2.4/string.py(205)safe_substitute()
-> return self.pattern.sub(convert, self.template)

and my diagnostic looked like this:

173 def subs(name):
174 vars[name] = string.Template(vars[name]).safe_substitute(vars)
175 try:
176 for name in 'subject', 'header_text', 'footer_text':
177 subs(name)
178 except UnicodeDecodeError, ude:
179 logger.exception(' name : %s exception : %s text %s ' % (name, ude, string.Template(vars[name])) )
180 raise

Revision history for this message
Sivan Greenberg (sivan) wrote :

I've done some debugging of my own, which reveals that the exception for me occurs in:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)
> /media/sdb1/debug-singing/collective.dancing/trunk-buildout/eggs/zope.i18n-3.6.0-py2.4.egg/zope/i18n/__init__.py(161)replace()
-> return unicode(mapping.get(param1 or param2, whole))

the piece of code is:

(Pdb) 156 u'This is ${name}'
157 """
158
159 def replace(match):
160 whole, param1, param2 = match.groups()
161 -> return unicode(mapping.get(param1 or param2, whole))
162
163 if not text or not mapping:
164 return text
165
166 return _interp_regex.sub(replace, text)

and I can confirm this happens only when the title is set with hebrew text

Revision history for this message
Daniel Nouri (daniel.nouri) wrote :

I tried to reproduce this, without much luck. I added a test in r78127.

I'm clueless about what's going on... What browser are you using to fill in the header and footer text?

Changed in singing-dancing:
assignee: nobody → daniel.nouri
Revision history for this message
Russ Ferriday (russf) wrote :

This text was created by the client, either in firefox or Safari, but definitely with text pasted from M$Word. That's where the Dash came from. EN DASH == \xe2\x80\x93

Revision history for this message
Daniel Nouri (daniel.nouri) wrote :

When I paste text using Firefox on a somewhat clean install, I get unicode in the header_text and footer_text variables. Thus, it's okay for me to paste non-ASCII characters.

Russ, can you reproduce yourself? Maybe this only happens with IE?

Revision history for this message
Russ Ferriday (russf) wrote : Re: [Bug 304456] Re: str type used in composer.py - causing UnicodeDecodeErrors

Daniel,
I've just checked, and IE may have been involved.
I only reproduced at the level of test text, not from pasting into
browser.
--r

On 30 Dec 2008, at 14:49, Daniel Nouri wrote:

> When I paste text using Firefox on a somewhat clean install, I get
> unicode in the header_text and footer_text variables. Thus, it's okay
> for me to paste non-ASCII characters.
>
> Russ, can you reproduce yourself? Maybe this only happens with IE?
>
> --
> str type used in composer.py - causing UnicodeDecodeErrors
> https://bugs.launchpad.net/bugs/304456
> You received this bug notification because you are a direct subscriber
> of the bug.

Russ Ferriday - Topia Systems - Open Source content management with
Plone and Zope
<email address hidden> - office: +44 2076 1777588 - mobile: +44 7789 338868 -
skype: ferriday

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.