Zope Content Management Framework (CMF)

GenericSetup export fails with non-ASCII properties

Reported by Tom Gross on 2008-06-24
4
Affects Status Importance Assigned to Milestone
Zope CMF
Undecided
Jens Vagelpohl

Bug Description

Exporting properties containing non-ASCII characters like "Ö" with GenericSetup "portal_setup" fails with:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.GenericSetup.tool, line 590, in manage_exportSelectedSteps
  Module Products.GenericSetup.tool, line 1011, in _doRunExportSteps
  Module Products.CMFPlone.exportimport.propertiestool, line 56, in exportPloneProperties
  Module Products.GenericSetup.utils, line 529, in _exportBody
  Module xml.dom.minidom, line 59, in toprettyxml
  Module Products.GenericSetup.utils, line 439, in writexml
  Module Products.GenericSetup.utils, line 410, in writexml
  Module Products.GenericSetup.utils, line 410, in writexml
  Module Products.GenericSetup.utils, line 410, in writexml
  Module Products.GenericSetup.utils, line 393, in writexml
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

This was seen on GenericSetup 1.3.3

It seems as if the refered value is encoded 'utf-8' already:

> /home/zope/buildouts/plone3/parts/plone/GenericSetup/utils.py(395)writexml()
-> a_value = escape(attrs[a_name].value.encode('utf-8'), quote=True)
(Pdb) attrs[a_name].value
'at|\xc3\x96sterreich'
(Pdb) unicode(attrs[a_name].value)
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
(Pdb) unicode(attrs[a_name].value, 'utf-8').encode('utf-8') == attrs[a_name].value
True
(Pdb)

Change line 663 of GenericSetup 1.4.0 (class:PropertyManagerHelpers, method:_extractProperties) from

value.decode(self._encoding)

into

value = value.decode(self._encoding)

Decoding is not done inplace!

The xml still doesn't look pretty, but it works at least.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tom Gross wrote:
> Public bug reported:
>
> Exporting properties containing non-ASCII characters like "Ö" with
> GenericSetup "portal_setup" fails with:
>
> Traceback (innermost last):
> Module ZPublisher.Publish, line 119, in publish
> Module ZPublisher.mapply, line 88, in mapply
> Module ZPublisher.Publish, line 42, in call_object
> Module Products.GenericSetup.tool, line 590, in manage_exportSelectedSteps
> Module Products.GenericSetup.tool, line 1011, in _doRunExportSteps
> Module Products.CMFPlone.exportimport.propertiestool, line 56, in exportPloneProperties
> Module Products.GenericSetup.utils, line 529, in _exportBody
> Module xml.dom.minidom, line 59, in toprettyxml
> Module Products.GenericSetup.utils, line 439, in writexml
> Module Products.GenericSetup.utils, line 410, in writexml
> Module Products.GenericSetup.utils, line 410, in writexml
> Module Products.GenericSetup.utils, line 410, in writexml
> Module Products.GenericSetup.utils, line 393, in writexml
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
>
> This was seen on GenericSetup 1.3.3
>
> It seems as if the refered value is encoded 'utf-8' already:
>
>> /home/zope/buildouts/plone3/parts/plone/GenericSetup/utils.py(395)writexml()
> -> a_value = escape(attrs[a_name].value.encode('utf-8'), quote=True)
> (Pdb) attrs[a_name].value
> 'at|\xc3\x96sterreich'
> (Pdb) unicode(attrs[a_name].value)
> *** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
> (Pdb) unicode(attrs[a_name].value, 'utf-8').encode('utf-8') == attrs[a_name].value
> True
> (Pdb)
>
> Change line 663 of GenericSetup 1.4.0 (class:PropertyManagerHelpers,
> method:_extractProperties) from
>
> value.decode(self._encoding)
>
> into
>
> value = value.decode(self._encoding)
>
> Decoding is not done inplace!
>
> The xml still doesn't look pretty, but it works at least.

Already fixed (though without a test) on the trunk. Likely needs both a
test and backporting to the 1.3 branch.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIYOEB+gerLs4ltQ4RArT4AKCyrJvi3ZzQgtFzeE+dzwrAXhP+NwCfVwET
WseqWMKw66NY1T7AeGeVd/w=
=NWCc
-----END PGP SIGNATURE-----

Tom Gross (itconsense) wrote :

This modified test_utils from GenericSetup 1.4.1 includes a test "test__extractProperties_encoded" verifying the issue. feel free to include it into the product. I don't have a commit access to the zope-repository.

Jens Vagelpohl (dataflake) wrote :
Changed in zope-cmf:
assignee: nobody → jens-dataflake
status: New → Fix Committed
Jens Vagelpohl (dataflake) wrote :

wrong status

Changed in zope-cmf:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments