Memory error with incomplete iterparse iteration

Bug #1737825 reported by Roma Klapaukh on 2017-12-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxml
High
scoder

Bug Description

When looping over the output of iterparse with a supplied schema, I get a memory error when python exits if the iteration was exited early.

The error is:
python3(42293,0x7fffb8c4c340) malloc: *** error for object 0x7fbfe0f03128: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

This error does not occur if a schema is not supplied to iterparse. It also does not occur when the breaking out of the loop happens. It happens when python eventually exits.

System information and minimal working example code + input files follow:

OS info:
$ uname -a
Darwin nix 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64

Requested information:
>>> print("%-20s: %s" % ('Python', sys.version_info))
Python : sys.version_info(major=3, minor=6, micro=3, releaselevel='final', serial=0)
>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
lxml.etree : (4, 1, 1, 0)
>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
libxml used : (2, 9, 5)
>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
libxml compiled : (2, 9, 5)
>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
libxslt used : (1, 1, 32)
>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
libxslt compiled : (1, 1, 29)

Files / code to reproduce:

test.py
-----
#!/usr/bin/env python3

from lxml.etree import parse, XMLSchema, iterparse

xml_file = 'test.xml'
xsd_file = 'library.xsd'

xsd_document = parse(xsd_file)
schema = XMLSchema(xsd_document)

context = iterparse(xml_file, schema=schema, events=['end'])
for _, elem in context:
    exit()

library.xsd
-----
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:d="http://test.com/library"
   xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"
   targetNamespace="http://test.com/library">
   <xs:element name="metadata" type="xs:string"/>
</xs:schema>

test.xml
--------
<?xml version="1.0"?>
<d:document xmlns:d="http://test.com/library" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://test.com/library">
  <d:metadata>Cats</d:metadata>
</d:document>

Roma Klapaukh (klapaukh) wrote :
scoder (scoder) wrote :

Thanks for the report. I can reproduce this. Might still be a problem in libxml2, but I'll investigate.

Changed in lxml:
assignee: nobody → scoder (scoder)
importance: Undecided → High
status: New → Confirmed
scoder (scoder) wrote :

Fixed here (test case added later):
https://github.com/lxml/lxml/commit/7affcdc3dc774cdf8db352890fc6cf538a70c01d

Thanks for the excellent reproducer, that made it easy to pin-point the problem.

Changed in lxml:
status: Confirmed → Fix Committed
scoder (scoder) on 2018-01-06
Changed in lxml:
milestone: none → 4.2.0
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers