XMLSchema failure with libxml2 2.9.4

[Thomas Klausner]

With pyhon-3.5.2 on NetBSD-7.99.34/amd64, there is a new self-test failure:

Traceback (most recent call last):
  File "/usr/pkg/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/pkg/lib/python3.5/unittest/case.py", line 600, in run
    testMethod()
  File "/scratch/textproc/py-lxml/work/lxml-3.6.1/src/lxml/tests/test_xmlschema.py", line 190, in test_xmlschema_parse_fixed_attributes
    parser=parser)
  File "/scratch/textproc/py-lxml/work/lxml-3.6.1/src/lxml/tests/common_imports.py", line 190, in parse
    return etree.parse(f, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:79840)
    doc = _parseDocument(source, parser, base_url)
  File "src/lxml/parser.pxi", line 1810, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:116398)
    return _parseMemoryDocument(source.getvalue(), url, parser)
  File "src/lxml/parser.pxi", line 1830, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:116673)
    c_doc = _parseDoc(text, url, parser)
  File "src/lxml/parser.pxi", line 1711, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:115219)
    return (<_BaseParser>parser)._parseUnicodeDoc(text, c_filename)
  File "src/lxml/parser.pxi", line 1051, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:109344)
    return context._handleParseResultDoc(self, result, None)
  File "src/lxml/parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103583)
    return _handleParseResult(self, self._c_ctxt, result,
  File "src/lxml/parser.pxi", line 694, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105237)
    _raiseParseError(c_ctxt, filename, context._error_log)
  File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:104146)
    raise error_log._buildParseException(
  File "<string>", line None
lxml.etree.XMLSyntaxError: Element 'b', attribute 'hardy': The value '(null)' does not match the fixed value constraint 'hey'.

Here's the requested output:

Python : sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
lxml.etree : (3, 6, 1, 0)
libxml used : (2, 9, 4)
libxml compiled : (2, 9, 4)
libxslt used : (1, 1, 29)
libxslt compiled : (1, 1, 29)

[Thomas Klausner (tk-giga)]

This bug still exists in 3.6.4.

[Daniel Santibáñez Polanco (dansanti)]

Did you upgrade from another ubuntu version?

in a clean version of ubuntu that works fine, but when i upgrated from ubuntu 14.04 xsd validation breaks, and can't get any value ( attribute or text)

[Thomas Klausner (tk-giga)]

As I wrote, I'm not on Ubuntu, but NetBSD.

[Alexander Pyhalov (alp-rsu)]

See the same on OpenIndiana with Python 2.7

TESTED VERSION: 3.5.0
    Python: sys.version_info(major=2, minor=7, micro=12, releaselevel='final', serial=0)
    lxml.etree: (3, 5, 0, 0)
    libxml used: (2, 9, 4)
    libxml compiled: (2, 9, 4)
    libxslt used: (1, 1, 28)
    libxslt compiled: (1, 1, 28)

======================================================================
ERROR: test_xmlschema_parse_fixed_attributes (lxml.tests.test_xmlschema.ETreeXMLSchemaTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/export/home/alp/srcs/oi-userland/components/python/pylxml/lxml-3.5.0/src/lxml/tests/test_xmlschema.py", line 190, in test_xmlschema_parse_fixed_attributes
    parser=parser)
  File "/export/home/alp/srcs/oi-userland/components/python/pylxml/lxml-3.5.0/src/lxml/tests/common_imports.py", line 190, in parse
    return etree.parse(f, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:85131)
    doc = _parseDocument(source, parser, base_url)
  File "src/lxml/parser.pxi", line 1799, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:124241)
    return _parseMemoryDocument(source.getvalue(), url, parser)
  File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:124533)
    c_doc = _parseDoc(text, url, parser)
  File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:123074)
    return (<_BaseParser>parser)._parseDoc(c_text, c_len, c_filename)
  File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:117114)
    return context._handleParseResultDoc(self, result, None)
  File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:110510)
    return _handleParseResult(self, self._c_ctxt, result,
  File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:112276)
    _raiseParseError(c_ctxt, filename, context._error_log)
  File "src/lxml/parser.pxi", line 613, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:111124)
    raise error_log._buildParseException(
XMLSyntaxError: Element 'b', attribute 'hardy': The value '(null)' does not match the fixed value constraint 'hey'.

[Mathy Vanvoorden (mathy)]

I came across the same bug while testing in Gentoo. I did some digging and found that this is because although the attribute_defaults=True flag is passed to libxml2, there's an issue with libxml2 not filling in the values for a schema.

I tried creating a patch but the only thing I can think of is to include the attribute hardy in all the elements and not rely on the autofill and I don't think that is the objective of this test function.

References I found going back to 2009 and earlier (by Stefan Behnel):

https://mail.gnome.org/archives/xml/2009-May/msg00036.html
https://bugzilla.gnome.org/show_bug.cgi?id=584385
https://bugs.launchpad.net/lxml/+bug/367942

I have no idea if this actually worked in between then and now or never worked. In any case, if libxml2 is not supporting it, perhaps this test / functionality should be disabled for now?

[Mathy Vanvoorden (mathy)]

After a bit more research I found that it might not even be that, if I disable the attribute_defaults flag and manually add the attributes, I still get the same error. Only completely removing the test case helps.

[Yves Chevallier (ychevallier)]

I can reproduce the bug with the following code:

```
In [21]: xsd = """<?xml version="1.0" encoding="utf-8"?>
    ...: <xs:schema version="1.0"
    ...: xmlns:xs="http://www.w3.org/2001/XMLSchema"
    ...: targetNamespace="http://schemas.foo.com/foo"
    ...: xmlns="http://schemas.foo.com/foo"
    ...: elementFormDefault="qualified">
    ...: <xs:element name="Symbols" type="Symbols"/>
    ...: <xs:complexType name="Symbols">
    ...: <xs:attribute name="version" type="xs:string" use="required" fixed="1.0"/>
    ...: </xs:complexType>
    ...: </xs:schema>"""
    ...:
    ...: xml = """<?xml version="1.0" ?>
    ...: <Symbols version="1.0" xmlns="http://schemas.foo.com/foo">
    ...: </Symbols>"""
    ...:
    ...: from lxml.etree import XMLSyntaxError, XMLSchema
    ...: from lxml import objectify
    ...:
    ...: open('xsd', 'w').write(xsd)
    ...: schema = XMLSchema(file='xsd')
    ...: parser = objectify.makeparser(schema=schema)
    ...: objectify.fromstring(xml, parser)
    ...:
  File "<string>", line unknown
XMLSyntaxError: Element '{http://schemas.foo.com/foo}Symbols', attribute 'version': The value '(null)' does not match the fixed value constraint '1.0'.
```

Version:

```
In [23]: lxml.etree.LXML_VERSION
Out[23]: (3, 7, 3, 0)
```

[Yves Chevallier (ychevallier)]

The bug is there since lxml>3.4.4

[Thomas Klausner (tk-giga)]

The line numbers have slightly changed, but the bug is still there in 3.8.0:

======================================================================
ERROR: test_xmlschema_parse_fixed_attributes (lxml.tests.test_xmlschema.ETreeXMLSchemaTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/pkg/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/scratch/textproc/py-lxml/work/lxml-3.8.0/src/lxml/tests/test_xmlschema.py", line 217, in test_xmlschema_parse_fixed_attributes
    parser=parser)
  File "/scratch/textproc/py-lxml/work/lxml-3.8.0/src/lxml/tests/common_imports.py", line 164, in parse
    return etree.parse(f, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3442, in lxml.etree.parse (src/lxml/lxml.etree.c:81716)
    doc = _parseDocument(source, parser, base_url)
  File "src/lxml/parser.pxi", line 1828, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118859)
    return _parseMemoryDocument(source.getvalue(), url, parser)
  File "src/lxml/parser.pxi", line 1848, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:119128)
    c_doc = _parseDoc(text, url, parser)
  File "src/lxml/parser.pxi", line 1729, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:117686)
    return (<_BaseParser>parser)._parseUnicodeDoc(text, c_filename)
  File "src/lxml/parser.pxi", line 1063, in lxml.etree._BaseParser._parseUnicodeDoc (src/lxml/lxml.etree.c:111673)
    return context._handleParseResultDoc(self, result, None)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105896)
    return _handleParseResult(self, self._c_ctxt, result,
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:107604)
    _raiseParseError(c_ctxt, filename, context._error_log)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:106458)
    raise error_log._buildParseException(
  File "<string>", line 0
lxml.etree.XMLSyntaxError: Element 'b', attribute 'hardy': The value '(null)' does not match the fixed value constraint 'hey'.

[scoder (scoder)]

There are XMLSchema related regressions in libxml2 2.9.4. Does anyone experience this problem with an earlier version? If not, a new libxml2 release is scheduled for the next weeks, so maybe someone can test with latest libxml2 master to see if it's fixed there?

[scoder (scoder)]

(Most) binary wheels of lxml 4.0.0 use libxml2 2.9.5 now.