Memory error with incomplete iterparse iteration

Bug #1737825 reported by Roma Klapaukh
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released

Bug Description

When looping over the output of iterparse with a supplied schema, I get a memory error when python exits if the iteration was exited early.

The error is:
python3(42293,0x7fffb8c4c340) malloc: *** error for object 0x7fbfe0f03128: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

This error does not occur if a schema is not supplied to iterparse. It also does not occur when the breaking out of the loop happens. It happens when python eventually exits.

System information and minimal working example code + input files follow:

OS info:
$ uname -a
Darwin nix 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64

Requested information:
>>> print("%-20s: %s" % ('Python', sys.version_info))
Python : sys.version_info(major=3, minor=6, micro=3, releaselevel='final', serial=0)
>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
lxml.etree : (4, 1, 1, 0)
>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
libxml used : (2, 9, 5)
>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
libxml compiled : (2, 9, 5)
>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
libxslt used : (1, 1, 32)
>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
libxslt compiled : (1, 1, 29)

Files / code to reproduce:
#!/usr/bin/env python3

from lxml.etree import parse, XMLSchema, iterparse

xml_file = 'test.xml'
xsd_file = 'library.xsd'

xsd_document = parse(xsd_file)
schema = XMLSchema(xsd_document)

context = iterparse(xml_file, schema=schema, events=['end'])
for _, elem in context:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:d=""
   xmlns:xs="" elementFormDefault="qualified"
   <xs:element name="metadata" type="xs:string"/>

<?xml version="1.0"?>
<d:document xmlns:d="" xmlns:xs="" xmlns:xsi="">

Revision history for this message
Roma Klapaukh (klapaukh) wrote :
Revision history for this message
scoder (scoder) wrote :

Thanks for the report. I can reproduce this. Might still be a problem in libxml2, but I'll investigate.

Changed in lxml:
assignee: nobody → scoder (scoder)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
scoder (scoder) wrote :

Fixed here (test case added later):

Thanks for the excellent reproducer, that made it easy to pin-point the problem.

Changed in lxml:
status: Confirmed → Fix Committed
scoder (scoder)
Changed in lxml:
milestone: none → 4.2.0
scoder (scoder)
Changed in lxml:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.