Premature end of data in tag file line 247651 with libxml2

[Igor Shishkin]

Hello,

I'm getting the following behaviour of createrepo on Ubuntu 14.04.1 LTS with libxml2=2.9.1+dfsg1-3ubuntu4.3(trusty-updates/main) and libxml2=2.9.1+dfsg1-3ubuntu4(trusty/main).

createrepo -g /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/comps.xml \
  -o /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/ /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/
I/O error : No such file or directory

(process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag package line 246828

(process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag filelists line 2

Spawning worker 0 with 54 pkgs
Spawning worker 1 with 54 pkgs
Spawning worker 2 with 54 pkgs
Spawning worker 3 with 54 pkgs
Spawning worker 4 with 54 pkgs
Spawning worker 5 with 54 pkgs
Spawning worker 6 with 54 pkgs
Spawning worker 7 with 54 pkgs
Spawning worker 8 with 54 pkgs
Spawning worker 9 with 54 pkgs
Spawning worker 10 with 53 pkgs
Spawning worker 11 with 53 pkgs
Spawning worker 12 with 53 pkgs
Spawning worker 13 with 53 pkgs
Spawning worker 14 with 53 pkgs
Spawning worker 15 with 53 pkgs
Spawning worker 16 with 53 pkgs
Spawning worker 17 with 53 pkgs
Spawning worker 18 with 53 pkgs
Spawning worker 19 with 53 pkgs
Spawning worker 20 with 53 pkgs
Spawning worker 21 with 53 pkgs
Spawning worker 22 with 53 pkgs
Spawning worker 23 with 53 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Traceback (most recent call last):
  File "/usr/share/createrepo/genpkgmetadata.py", line 308, in <module>
    main(sys.argv[1:])
  File "/usr/share/createrepo/genpkgmetadata.py", line 280, in main
    mdgen.doRepoMetadata()
  File "/usr/lib/python2.7/dist-packages/createrepo/__init__.py", line 999, in doRepoMetadata
    rp.getFilelists(complete_path, csum)
  File "/usr/lib/python2.7/dist-packages/sqlitecachec.py", line 55, in getFilelists
    self.repoid))
TypeError: Parsing filelists.xml error: Premature end of data in tag file line 247651

Problem fixes by installing libxml2=2.9.1+dfsg1 from debian sid.
Comparing patchsets in debian package sources between 2.9.1+dfsg1 and 2.9.1+dfsg1-3ubuntu4.3 I see in debian package much more patches, I suppose one of them helps in my problem:

2.9.1+dfsg1-3ubuntu4.3:

0001-modify-xml2-config-and-pkgconfig-behaviour.patch
0002-Fix-an-error-in-xmlCleanupParser.patch
0003-Fix-missing-break-on-last-function-for-attributes.patch
0004-xmllint-memory-should-fail-on-empty-files.patch
0005-properly-quote-the-namespace-uris-written-out-during.patch
0006-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch
0006-fix-python-multiarch-includes.patch
0007-Fix-XPath-optimization-with-predicates.patch
CVE-2014-0191.patch
lp1321869.patch
xmllint_pretty.patch

2.9.1+dfsg1:

0001-modify-xml2-config-and-pkgconfig-behaviour.patch
0002-fix-python-multiarch-includes.patch
0003-Fix-an-error-in-xmlCleanupParser.patch
0004-Fix-missing-break-on-last-function-for-attributes.patch
0005-xmllint-memory-should-fail-on-empty-files.patch
0006-properly-quote-the-namespace-uris-written-out-during.patch
0007-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch
0008-missing-else-in-xlink.c.patch
0009-Catch-malloc-error-and-exit-accordingly.patch
0010-Fix-handling-of-mmap-errors.patch
0011-Avoid-crash-if-allocation-fails.patch
0012-Fix-a-possible-NULL-dereference.patch
0013-Clear-up-a-potential-NULL-dereference.patch
0014-Fix-XPath-optimization-with-predicates.patch
0015-xmllint-pretty-crashed-without-following-numeric-arg.patch
0016-Fix-potential-NULL-pointer-dereferences-in-regexp-co.patch
0017-Fix-a-potential-NULL-dereference-in-tree-code.patch
0018-Fix-pointer-dereferenced-before-null-check.patch
0019-Fix-a-bug-loading-some-compressed-files.patch
0020-Avoid-a-possibility-of-dangling-encoding-handler.patch
0021-Fix-a-couple-of-missing-NULL-checks.patch
0022-adding-init-calls-to-xml-and-html-Read-parsing-entry.patch
0023-Handling-of-XPath-function-arguments-in-error-case.patch
0024-Missing-initialization-for-the-catalog-module.patch
0025-Fix-an-fd-leak-in-an-error-case.patch
0026-fixing-a-ptotential-uninitialized-access.patch
0027-Fix-xmlTextWriterWriteElement-when-a-null-content-is.patch
0028-Avoid-a-possible-NULL-pointer-dereference.patch
0029-Do-not-fetch-external-parameter-entities.patch
0030-Avoid-Possible-null-pointer-dereference-in-memory-de.patch
0031-xmllint-was-not-parsing-the-c14n11-flag.patch
0032-Fix-regressions-introduced-by-CVE-2014-0191-patch.patch

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Quanah Gibson-Mount (mishikal)]

When will this be fixed? I'm currently using a single server to host both debs and rpms, and this bug triggers fairly frequently, causing me to have to take down my repo and regenerate it.

[Sebastien Bacher (seb128)]

Thank you for your bug report. Do you have a testcase you could add to the bug? That would be make easier to test/find out the fix to backport

Aron, you seem to co-maintain libxml in Debian, do you have any idea about the issue?

[Quanah Gibson-Mount (mishikal)]

Hi Sebastien,

The problem is I've found no consistent trigger for the bug. I have 225 packages in my repository, built out for two different branches and two oses (so 225x4). createrepo just randomly corrupts when executing 'createrepo --update .' when adding a new source/binary rpm pair. It is trivially solvable (rm -rf repodata .repodata; createrepo .), but having to briefly have a 100% repo outage is the problem.

I've not seen anything specific about the packages that would cause this to occur. It seems to be some type of race condition during the update process, because of it complaining about " I/O error : No such file or directory".

[Lenz Grimmer (lenzgr)]

FWIW, I am currently experiencing the same error when running "createrepo" on a repo containing 20 RPMs. The host system is running Ubuntu 14.04.5 LTS, libxml2 version is 2.9.1+dfsg1-3ubuntu. For me, the error occurs every time, removing the repodata directories does not resolve this.

 % createrepo .
Spawning worker 0 with 20 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
I/O error : No such file or directory

(process:18005): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag package line 2263

(process:18005): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag filelists line 2

Traceback (most recent call last):
  File "/usr/share/createrepo/genpkgmetadata.py", line 308, in <module>
    main(sys.argv[1:])
  File "/usr/share/createrepo/genpkgmetadata.py", line 280, in main
    mdgen.doRepoMetadata()
  File "/usr/lib/python2.7/dist-packages/createrepo/__init__.py", line 999, in doRepoMetadata
    rp.getFilelists(complete_path, csum)
  File "/usr/lib/python2.7/dist-packages/sqlitecachec.py", line 54, in getFilelists
    self.repoid))
TypeError: Parsing filelists.xml error: Premature end of data in tag file line 2332

[Brian C (briancl)]

I have seen a similar issue using 'yum check-update' on Ubunty Trusty with libxml2=2.9.1+dfsg1-3ubuntu4.8 failing to parse the primary.xml.gz index.

$ yum check-update

custom-repo: [########################################################################### ] 6900/6903
(process:1482): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: attributes construct error

(process:1482): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Couldn't find end of Start Tag size line 238347

(process:1482): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Premature end of data in tag package line 238338

(process:1482): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: Parsing primary.xml error: Premature end of data in tag metadata line 2

Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 288, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 140, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 436, in doCommands
    self._getTs(needTsRemove)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 101, in _getTs
    self._getTsInfo(remove_only)
  File "/usr/lib/python2.7/dist-packages/yum/depsolve.py", line 112, in _getTsInfo
    pkgSack = self.pkgSack
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 892, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/dist-packages/yum/__init__.py", line 673, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/dist-packages/yum/repos.py", line 294, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.7/dist-packages/yum/yumRepo.py", line 190, in populate
    dobj = repo_cache_function(xml, csum)
  File "/usr/lib/python2.7/dist-packages/sqlitecachec.py", line 46, in getPrimary
    self.repoid))
TypeError: Parsing primary.xml error: Specification mandate value for attribute archive

I have seen this a few times, but never took the time to dig into it until recently. What I found was the size of the primary.xml.gz file is what seems to be causing the failure. If (size of file in bytes)/(8192 bytes) has a remainder of 1 byte this failure will ...