Doc omission: XMLParser encoding is an iconv encoding name, not a Python encoding name

Bug #1842387 reported by Thomas ten Cate
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxml
Fix Released
Low
Unassigned

Bug Description

The documentation on https://lxml.de/api/lxml.etree.XMLParser-class.html only says:

> encoding - override the document encoding

It doesn't specify what encodings are valid, so it's reasonable to assume that it's the [list supported by Python](https://docs.python.org/3/library/codecs.html#standard-encodings). However, experimentation reveals that that's not the case:

>>> lxml.etree.XMLParser(encoding='utf_8_sig')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/lxml/parser.pxi", line 1518, in lxml.etree.XMLParser.__init__ (src/lxml/etree.c:117909)
  File "src/lxml/parser.pxi", line 821, in lxml.etree._BaseParser.__init__ (src/lxml/etree.c:110544)
LookupError: unknown encoding: 'b'utf_8_sig''

From what I can tell, the encoding must be one of the ones reported by `iconv -l`, which is surprising to Python developers, so it should at least be documented.

If there's some way to accept Python encodings here too, then that would of course be even better.

Experimentally I found that `utf-8` has the effect I intended (ignoring any UTF-8 encoded BOM), but `utf8` without a hyphen fails:

>>> lxml.etree.parse(io.BytesIO(b'\xef\xbb\xbf<abc/>'), parser=lxml.etree.XMLParser(encoding='utf8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/lxml/etree.pyx", line 3444, in lxml.etree.parse (src/lxml/etree.c:83185)
  File "src/lxml/parser.pxi", line 1851, in lxml.etree._parseDocument (src/lxml/etree.c:120981)
  File "src/lxml/parser.pxi", line 1871, in lxml.etree._parseMemoryDocument (src/lxml/etree.c:121250)
  File "src/lxml/parser.pxi", line 1759, in lxml.etree._parseDoc (src/lxml/etree.c:119926)
  File "src/lxml/parser.pxi", line 1125, in lxml.etree._BaseParser._parseDoc (src/lxml/etree.c:114173)
  File "src/lxml/parser.pxi", line 598, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/etree.c:107738)
  File "src/lxml/parser.pxi", line 709, in lxml.etree._handleParseResult (src/lxml/etree.c:109447)
  File "src/lxml/parser.pxi", line 638, in lxml.etree._raiseParseError (src/lxml/etree.c:108301)
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

I'm not sure what the deal is there, and it may be a separate issue.

Version info:

Python : sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
lxml.etree : (4, 1, 1, 0)
libxml used : (2, 9, 9)
libxml compiled : (2, 9, 8)
libxslt used : (1, 1, 33)
libxslt compiled : (1, 1, 32)

description: updated
description: updated
Revision history for this message
scoder (scoder) wrote :

Right. I'll add a docstring note. Thanks.

Changed in lxml:
importance: Undecided → Low
milestone: none → 5.0
status: New → Fix Committed
scoder (scoder)
Changed in lxml:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.