UnicodeDecodeError on sending content and visualizing "Statistics" page

[Andrea Russo]

Hi,

I tried to send a page content from my site and got an error. After that error, whenever I go to the Statistics page I get the error attached here. Maybe the problem is caused by accented characters?

Thanks for your time,
Andrea.

[Daniel Nouri (daniel.nouri)]

Andrea,

can you try and change your user interface language? The error log looks like it's failing while it tries to translate parts of the user interface. Is there a buggy PO file involved, maybe?

[Andrea Russo (andrea-russo)]

Dear Daniel,

thank you for your reply.

I tried to change laguage but the problem seems to be linked to the
content of a particular page in my site. You can see the page contents
below.

I attached also the error full trace.

If I can do other tests to help debugging this problem, please let me
know.

Thank you very much for your help.

Page Contents:

title: Attività del Focal Point Nazionale IPCC
description: Le principali attività del Focal Point Nazionale IPCC

<p>Le principali attività del Focal Point Nazionale IPCC includono:</p>
<ul><li>la raccolta di informazioni e la documentazione delle attività tecnico-scientifiche nazionali inerenti le tematiche trattate dall’IPCC: scienza del clima e cambiamenti climatici (osservazioni, modelli, studi di vulnerabilità, stima di impatti, misure di adattamento e mitigazione);&nbsp;</li><li>la divulgazione dell’attività e dei risultati dell’IPCC a tutti i livelli sul territorio nazionale tramite sito-web, riunioni, convegni e workshop dedicati;</li><li>l’individuazione e designazione degli esperti o i rappresentanti nazionali alle riunioni, workshop ed alle sessioni plenarie dell' IPCC e dei relativi gruppi di lavoro;</li><li>il riscontro all’IPCC sulle attività di rilievo realizzate sul territorio nazionale.</li></ul>
<p>L’attività di Focal Point IPCC per l’Italia è svolta dal <a title="Contatti" class="internal-link" href="contatti">Dr. Sergio Castellari</a> presso il Centro Euro-Mediterraneo per i Cambiamenti Climatici (CMCC) a partire dal 1 Dicembre 2006. <br />La documentazione precedente a tale data è reperibile sul sito a cura del precedente responsabile Focal Point IPCC Italia, Dr. Vincenzo Ferrara presso <a class="external-link" href="http://clima.casaccia.enea.it/ipcc/focalpoint/">ENEA-Casaccia</a>.</p>
<p>&nbsp;</p>

[Andrea Russo (andrea-russo)]

Hi,

I've got a little time to investigate further and now I can be more
precise about the bug I'm experiencing.

It seems that the problem is in the accented letter in the title of
the Plone document I pasted in the previous message.

In fact, just for try, I commented this line in page template
`collective.dancing-0.8.2-py2.4.egg/collective/dancing/browser/jobs.pt':

      <td tal:content="job/title">Send "My news item"</td>

and the exception when I visit the statistics page isn't raised
anymore.

I'm not able to find the right place in the code to fix this, but I
hope this information is of some help.

Thank you and best regards,
Andrea.

[Sivan Greenberg (sivan)]

I'm experiencing the same bug, this time after queuing a hebrew page to be sent from the statistics page, this is the traceback:

2008-12-02 15:20:11 ERROR Zope.SiteErrorLog http://zeta:8080/cancer/portal_newsletters/channels/newsletter-stats.html
Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module plone.z3cform.layout, line 49, in __call__
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
  Module Products.PageTemplates.PageTemplate, line 89, in pt_render
  Module zope.pagetemplate.pagetemplate, line 117, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 891, in do_useMacro
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 891, in do_useMacro
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 949, in do_defineSlot
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 957, in do_defineSlot
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 891, in do_useMacro
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 957, in do_defineSlot
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 534, in do_optTag_tal
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 949, in do_defineSlot
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 536, in do_optTag_tal
  Module zope.tal.talinterpreter, line 521, in do_optTag
  Module zope.tal.talinterprete...

[Sivan Greenberg (sivan)]

I can confirm commenting <td tal:content="job/title">Send "My news item"</td> fixes the problem

[Sivan Greenberg (sivan)]

I've done some debugging of my own, which reveals that the exception for me occurs in:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)
> /media/sdb1/debug-singing/collective.dancing/trunk-buildout/eggs/zope.i18n-3.6.0-py2.4.egg/zope/i18n/__init__.py(161)replace()
-> return unicode(mapping.get(param1 or param2, whole))

the piece of code is:

(Pdb) 156 u'This is ${name}'
157 """
158
159 def replace(match):
160 whole, param1, param2 = match.groups()
161 -> return unicode(mapping.get(param1 or param2, whole))
162
163 if not text or not mapping:
164 return text
165
166 return _interp_regex.sub(replace, text)

and I can confirm this happens only when the title is set with hebrew text

[tmog (mogensen)]

The UnicodeDecodeError on "Statistics" is indeed the result of a bad message in job.title when the content being sent has a non-ascii title.

I've committed a fix in r76770.

[Andrea Russo (andrea-russo)]

Hi,

I can confirm that the fix is working here.

Thank you very much for your help.

Regards,
Andrea.

[Daniel Widerin (saily)]

thanks tmog,
this is part of collective.dancing 0.8.3 and collective.singing 0.6.7