Memory error with incomplete iterparse iteration

[Roma Klapaukh]

When looping over the output of iterparse with a supplied schema, I get a memory error when python exits if the iteration was exited early.

The error is:
python3(42293,0x7fffb8c4c340) malloc: *** error for object 0x7fbfe0f03128: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

This error does not occur if a schema is not supplied to iterparse. It also does not occur when the breaking out of the loop happens. It happens when python eventually exits.

System information and minimal working example code + input files follow:

OS info:
$ uname -a
Darwin nix 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64

Requested information:
>>> print("%-20s: %s" % ('Python', sys.version_info))
Python : sys.version_info(major=3, minor=6, micro=3, releaselevel='final', serial=0)
>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
lxml.etree : (4, 1, 1, 0)
>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
libxml used : (2, 9, 5)
>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
libxml compiled : (2, 9, 5)
>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
libxslt used : (1, 1, 32)
>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
libxslt compiled : (1, 1, 29)

Files / code to reproduce:

test.py
-----
#!/usr/bin/env python3

from lxml.etree import parse, XMLSchema, iterparse

xml_file = 'test.xml'
xsd_file = 'library.xsd'

xsd_document = parse(xsd_file)
schema = XMLSchema(xsd_document)

context = iterparse(xml_file, schema=schema, events=['end'])
for _, elem in context:
    exit()

library.xsd
-----
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:d="http://test.com/library"
   xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified"
   targetNamespace="http://test.com/library">
   <xs:element name="metadata" type="xs:string"/>
</xs:schema>

test.xml
--------
<?xml version="1.0"?>
<d:document xmlns:d="http://test.com/library" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://test.com/library">
  <d:metadata>Cats</d:metadata>
</d:document>

[scoder (scoder)]

Thanks for the report. I can reproduce this. Might still be a problem in libxml2, but I'll investigate.

[scoder (scoder)]

Fixed here (test case added later):
https://github.com/lxml/lxml/commit/7affcdc3dc774cdf8db352890fc6cf538a70c01d

Thanks for the excellent reproducer, that made it easy to pin-point the problem.