Page templates messes up with the encoding of the title attribute in the ZMI

[Laurent Regnier]

Zope 2.10.5, Python 2.4.4

Summary: when the titles of folders and/or page templates contain non-ascii
caracters, weird things (unicode errors, incorrect display of strings...)
happen in the ZMI; the setting of the management_page_charset property can fix
some but not all of the problems.

Now let's go for the long version. I did the following:

1. create a new zope instance (mkzopeinstance.py), launch the zope server then
   log in the ZMI;

2. give a title to the root folder containing non-ascii caracters;

3. try any action involving ZPT (create a new ZPT, access the error log, edit
   /standard_page_template.pt): this always results in the same
   UnicodeDecodeError (see traceback below);

4. set the management_page_charset property of the root folder to utf-8. At
   this point I have no more unicode errors when accessing the ZPT management
   pages but I still have some problems:

   - in the Properties tab of a ZPT I get a UnicodeDecodeError if I try to save
     a title containing non-ascii caracters. Strangely enough this error is not
     recorded in the event log, so I can't produce a traceback. If I try to set
     the title from the Edit tab there is no problem.

   - if the ZPT's title is ascii I can save the property sheet, but properties
     having a non-ascii value are incorrectly displayed in the Properties
     tab.

   - if a folder has ZPT's and subfolders and if the titles of the subfolders
     contain non-ascii caracters then the titles of the ZPT's are incorrectly
     displayed in the folder Contents tab;

By "incorrectly displayed" I mean that the displayed string (value of the
property in the properties tab or title of the ZPT in the folder view) is
encoded as if it had been decoded from iso-8859-15, e.g. the word "étrange"
shows up as "étrange" (I checked that my browser (firefox) uses the right
encoding for displaying pages and for sending data in POST requests).

Outside the ZMI everything is fine: titles and properties are correctly encoded
and displayed in published pages.

I've unsuccesfully tried to reproduce (some of) these comportements in the
python interface (zopectl debug).

I've also tried to set the management_page_charset value to iso-8859-15; things
are a bit better, in particular everything is displayed correctly but I still
can't save the title of a ZPT containing non-ascii caracters, neither from the
Edit nor from the Properties tab. Sometimes I also get an error when viewing
the Properties tab of a zpt, but I couldn't figure out what were the exact
conditions to reproduce it (see traceback).

I haven't played with the UnicodeEncodingConflict machinery as I don't quite
understand how it interacts with the rest; is there any place where I could
find some doc on this?

-------------------------------------------------------------------------------
Traceback obtained when the management_page_charset property is not set and the
containing folder (/) has a title containing non-ascii characters:

2008-03-09T13:35:50 ERROR Zope.SiteErrorLog http://localhost:8080/standard_template.pt/pt_editForm
Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Shared.DC.Scripts.Bindings, line 313, in __call__
  Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
  Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
  Module Products.PageTemplates.PageTemplate, line 89, in pt_render
  Module zope.pagetemplate.pagetemplate, line 117, in pt_render
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 534, in do_optTag_tal
  Module zope.tal.talinterpreter, line 516, in no_tag
  Module zope.tal.talinterpreter, line 346, in interpret
  Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position 198: ordinal not in range(128)

-------------------------------------------------------------------------------
Traceback obtained when management_page_charset is set to iso-8859-15 and I try
to save a non-ascii title.

2008-03-09T15:34:29 ERROR Zope.SiteErrorLog http://localhost:8080/standard_template.pt/pt_editAction
Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.PageTemplates.ZopePageTemplate, line 192, in pt_editAction
  Module Products.PageTemplates.ZopePageTemplate, line 208, in pt_setTitle
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 4-6: invalid data

-------------------------------------------------------------------------------
Traceback that I occasionnaly obtain when management_page_charset is set
to iso-8859-15, but I can't figure out how to reproduce this error for sure.

2008-03-09T13:43:17 ERROR Zope.SiteErrorLog http://localhost:8080/standard_templ
ate.pt/manage_propertiesForm
Traceback (innermost last):
  Module ZPublisher.Publish, line 122, in publish
  Module ZServer.HTTPResponse, line 262, in setBody
  Module ZPublisher.HTTPResponse, line 324, in setBody
  Module ZPublisher.HTTPResponse, line 476, in _encode_unicode
  Module encodings.iso8859_15, line 18, in encode
UnicodeEncodeError: 'charmap' codec can't encode character u'\xa8' in position 11383: character maps to <undefined>

[Colin Watson (cjwatson)]

The zope2 project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope2.