lxml

tostring of single element also returns rest of whole document (when xhmtl1 dtd is declared)

Bug #1970741 reported by Wolfgang Schnerring on 2022-04-28

This bug affects 6 people

Affects		Status	Importance	Assigned to	Milestone
	lxml	Confirmed	High	Unassigned

Bug Description

I've discovered a strange edge case bug starting in lxml-4.7.1 (in 4.6.5 it's fine): When a document declares an xhtml1 DTD (but not other DTDs), etree.tostring() of a single element returns not only that element, but everything that comes after as well (which then is not even well-formed XML). Here's a reproduction recipe:

import lxml.etree
html = """\
<!DOCTYPE anything PUBLIC "anything" "{dtd}">
<html>
  <body>
    <div id="one">one</div>
    <div id="two">two</div>
  </body>
</html>"""
div = '<div id="one">one</div>'

for dtd in [
        'http://www.w3.org/TR/html401/strict.dtd',
        'http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd',
        'http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd',
        'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd']:
    doc = lxml.etree.fromstring(html.format(dtd=dtd))
    el = doc.getchildren()[0].getchildren()[0]
    text = lxml.etree.tostring(el, encoding=str)
    if text.strip() == div:
        print('PASS %s' % dtd)
    else:
        print('FAIL %s:\n%s' % (dtd, text))

With 4.7.1 and 4.8.0 this is the resulting output (4.6.5 gives 4xPASS):
PASS http://www.w3.org/TR/html401/strict.dtd
PASS http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd
FAIL http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd:
<div id="one">one</div>
<div id="two">two</div>
</body>
</html>

FAIL http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd:
<div id="one">one</div>
<div id="two">two</div>
</body>
</html>

Here's the detailed version info:
>>> print("%-20s: %s" % ('Python', sys.version_info))
Python : sys.version_info(major=3, minor=9, micro=12, releaselevel='final', serial=0)
>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
lxml.etree : (4, 8, 0, 0)
>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
libxml used : (2, 9, 12)
>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
libxml compiled : (2, 9, 12)
>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
libxslt used : (1, 1, 34)
>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
libxslt compiled : (1, 1, 34)