UnicodeEncodeError when parse XML with unicoded tags names

[petrikoz]

INFORMATION:

Python : sys.version_info(major=2, minor=7, micro=6, releaselevel='final', serial=0)
lxml.etree : (3, 4, 1, 0)
libxml used : (2, 9, 1)
libxml compiled : (2, 9, 1)
libxslt used : (1, 1, 28)
libxslt compiled : (1, 1, 28)

When parse XML file with utf-8 encoding and unicoded tags name have UnicodeEncodeError.

Code for illustration bug:

> import urllib2
> from lxml import etree

> xml_file = urllib2.urlopen('https://gist.githubusercontent.com/petrikoz/b5307616bab6247a958a/raw/81a52e2079986e57ab227e72e01b2e7b6f8d97ee/lxml-bug.xml')
> root = etree.XML(xml_file.read())
> root
> <repr(<lxml.etree._Element at 0x7f59f0733488>) failed: UnicodeEncodeError: 'ascii' codec can't encode characters in position 9-25: ordinal not in range(128)>

Why is it so?
Why lxml not returned unicoded representation of tag with unicoded name?

[scoder (scoder)]

My guess is that it's a Py2.x problem. IIRC, repr() is expected to return a byte string in Py2.x, and lxml returns a unicode string. Python then fails to encode it to a byte string. So the error happens outside of lxml, inside of Python. This has been fixed in Python 3.x, which properly supports (and in fact requires) a unicode text string as result of repr().

Given that the tag name may not be representable with an ASCII encoded byte string (and clearly is not in this case), there isn't really a correct way to do this. I mean, lxml could return something like "unprintable tag name" for non-ascii tag names in repr(), but that wouldn't really be satisfactory... Although it would still be better than letting Python raise an exception.

[petrikoz (po-zelenin)]

Ok. Thank you for your answer.

I agree what would be better letting Python raise an exception.

[Olli Pottonen (olli-pottonen)]

I think this is fixed on master on github, see https://github.com/lxml/lxml/pull/159.
But the sample xml is not available anymore, so I could not test and find out for certain.