make_links_absolute fails on bad links

Bug #1250557 reported by sylvain zimmer on 2013-11-12
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

make_links_absolute() is not robust enough when receiving bad input.

2 examples:

    <h1>Some test</h1>

    <!-- Links like this make lxml panic -->
    <a href="]Buy">test2</a>

This will throw

  File "/home/worker/code/venv/local/lib/python2.7/site-packages/lxml/html/", line 316, in make_links_absolute
  File "/home/worker/code/venv/local/lib/python2.7/site-packages/lxml/html/", line 437, in rewrite_links
    new_link = link_repl_func(link.strip())
  File "/home/worker/code/venv/local/lib/python2.7/site-packages/lxml/html/", line 315, in link_repl
    return urljoin(base_url, href)
  File "/usr/lib/python2.7/", line 260, in urljoin
    urlparse(url, bscheme, allow_fragments)
  File "/usr/lib/python2.7/", line 142, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib/python2.7/", line 190, in urlsplit
    raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL

Other example:

    <h1>Some a\x12b\x13c\x14d\x15e test</h1>
    <a href="/bad-link-a\x12b\x13c\x14d\x15e">test</a>

Will throw

  File "/home/worker/code/venv/local/lib/python2.7/site-packages/lxml/html/", line 316, in make_links_absolute
  File "/home/worker/code/venv/local/lib/python2.7/site-packages/lxml/html/", line 454, in rewrite_links
    el.attrib[attrib] = new_link
  File "lxml.etree.pyx", line 2222, in lxml.etree._Attrib.__setitem__ (src/lxml/lxml.etree.c:54583)
  File "apihelpers.pxi", line 520, in lxml.etree._setAttributeValue (src/lxml/lxml.etree.c:17678)
  File "apihelpers.pxi", line 1335, in lxml.etree._utf8 (src/lxml/lxml.etree.c:24701)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters

It would be great to just ignore these malformed links or at least have an option to do so, because for now my only choice is to disable make_links_absolute() entirely or do some pre-processing that would make it useless in the first place.

Python : sys.version_info(major=2, minor=7, micro=2, releaselevel='final', serial=0)
lxml.etree : (3, 2, 4, 0)
libxml used : (2, 7, 2)
libxml compiled : (2, 7, 2)
libxslt used : (1, 1, 28)
libxslt compiled : (1, 1, 28)

Thanks a lot!

scoder (scoder) wrote :

The first part is fixed by adding a new option "handle_failures" here:

The second part is more tricky. The right fix would be to generally handle encoding problems in parsed broken HTML trees better.

Thanks a lot for the first part of the fix :)

scoder (scoder) on 2013-11-15
Changed in lxml:
status: New → Confirmed
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers