lxml.html interface for iterparse?

[danny0838]

I am parsing an HTML using iterparse. However, the returned element in each iteration does not implement the interface of lxml.html (https://lxml.de/lxmlhtml.html#html-element-methods), such as .classes, .text_content(), etc., even if html=True parameter is passed to iterparse.

Is there a way to parse an HTML iteratively like iterparse with lxml.html interface supported?

[scoder (scoder)]

Seems a reasonable addition that should be trivial to add. PR welcome. Tests can go into a new file "test_parser.py" in "src/lxml/html/tests/".

[danny0838 (danny0838)]

What should the right interface for it to be? Should it be lxml.html.iterparse()?

[scoder (scoder)]

> lxml.html.iterparse()?

Yes.

[danny0838 (danny0838)]

OK. Unfortunately I'm not familiar with C programming and probably can't contribute the code.

[scoder (scoder)]

Does this look like C?
https://github.com/lxml/lxml/blob/master/src/lxml/html/__init__.py

[danny0838 (danny0838)]

Unlike etree.fromstring(), etree.iterparse() does not support the "parser" argument, and thus lxml.html.iterparse() cannot be simply implemented as lxml.html.fromstring(), which simply extends the passed "parser" argument. It seems that some tweaks of etree.pyx or iterparse.pxi is required, which at least require some Python C module programming knowledge.

[scoder (scoder)]

You can add a new keyword argument "element_class_lookup" to iterparse():
https://github.com/lxml/lxml/blob/master/src/lxml/iterparse.pxi

I trust that a little bit of Python knowledge is sufficient for this.

Tests can go into src/lxml/html/tests/test_basic.py, but it seems worth adding a new unittest test class for it there. Or a new doctest file, if you prefer that.