ubuntu-docs build failure - possible solution

[Gunnar Hjalmarsson]

ubuntu-docs failed to build during the second Focal Fossa test rebuild:

https://launchpad.net/ubuntu/+archive/test-rebuild-20200327-focal/+build/18982738

It was itstool which failed to generate the localized zh_CN pages. If dropping zh_CN from ubuntu-help/Makefile.am, ubuntu-docs builds. But excluding the Chinese pages is apparently not an acceptable solution.

The itstool package in focal {build-,}depends on python3. I built itstool in a PPA with python2 instead, and in that PPA ubuntu-docs built fine without modifications:

https://launchpad.net/~gunnarhj/+archive/ubuntu/ubuntu-docs/+packages

So the root cause seems to lie neither in ubuntu-docs nor itstool. I would suspect that python3-libxml2 is buggy, but since itstool was built against python2 in eoan, it's hard to tell if it's a regression.

Would building itstool against python2 be an acceptable workaround in focal?

[Gunnar Hjalmarsson (gunnarhj)]

@Boyuan: Do you have any thoughts on the above?

[Gunnar Hjalmarsson (gunnarhj)]

Hmm.. I saw in the buildlog from the successful build that it skipped the Chinese translation of a particular string. The reason proved to be a missing '<' character, i.e. the translated string contains an instance of '/em>' instead of '</em>'. And that's the correct behavior; it should check the Mallard syntax and skip translations with invalid syntax.

Anyway, I added the missing '<', and then ubuntu-docs built with the itstool version in focal.

So the direct reason for the build failure is a syntactically incorrect translation. OTOH, translators make mistakes, and a bad translation *should not* make the whole package build to fail.

[Gunnar Hjalmarsson (gunnarhj)]

@Shaun: Have you possibly stumbled into anything similar when building e.g. gnome-user-docs?

[Gunnar Hjalmarsson (gunnarhj)]

Uploaded a new version of ubuntu-docs, which builds on focal despite the issue discussed in this bug report.

[Sebastien Bacher (seb128)]

It's segfault in libxml code, their binding doesn't seem to handle invalid files well, somewaht similar to https://gitlab.gnome.org/GNOME/libxml2/-/issues/12

[Boyuan Yang (hosiet)]

@Gunnar It seems that the root cause has been found, thanks! It's well known that translation could be fragile when the build system is not well defined; all I can think of is to make the build system robust enough and yell out loudly enough whenever an error (e.g., segfault) happens.

[Gunnar Hjalmarsson (gunnarhj)]

Thanks Sebastien!

Or https://gitlab.gnome.org/GNOME/libxml2/issues/64 (which is not yet closed).

@Boyuan: Well, it does scream loudly in the sense that the build fails.

I wouldn't blame the build system in this case. It has worked in a sensible and robust way for many years. The segfault issue only happens with python3-libxml2, not with python-libxml2. And I have the impression (not yet confirmed) that the segfault only happens for XML syntax errors in Chinese translations; for similar errors in non-Chinese translations it only prints a warning and moves on (which is the desired behavior).

[Shaun McCance (shaunm-gnome)]

So in general, syntax errors in translations just cause a warning, and the original text is used. itstool only errors on syntax if you pass --strict, and yelp.m4 doesn't even give you a way to pass --strict.

There's a chance this is itstool's fault. You do still have to do memory management with libxml2's Python bindings, so stuff like double frees in application code is possible. But since this crash happens during error reporting for a language with many-byte UTF-8 chars, and since it weirdly only happens with Python 3, I suspect it's this libxml2 bug:

https://bugzilla.gnome.org/show_bug.cgi?id=789714

[Gunnar Hjalmarsson (gunnarhj)]

Thanks Shaun! It's indeed that bug. Why am I not surprised when seeing that you filed it? :)

I grabbed a Fedora patch and submitted a Debian merge request:

https://salsa.debian.org/xml-sgml-team/libxml2/-/merge_requests/3

It looks pretty spot on, don't you think? At least, with that patch applied I can build ubuntu-docs 19.10.2 successfully again. A syntax error in one of the Chinese translations is reported as warnings as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-docs - 20.04.1

---------------
ubuntu-docs (20.04.1) focal; urgency=medium

  * scripts/update-translations.sh:
    - Handle tarballs also from the package
  * Bump version to 20.04 / focal
  * debian/control.in, debian/control:
    - Replace Vcs-Bzr with Vcs-Git
    - Bump Standards-Version to 4.5.0
  * New languages: af, ckb
  * Translations from Launchpad:
    - Syntactically incorrect Chinese translations of two strings were
      corrected before this translation update, which works around the
      focal build failure as described in LP: #1869814.

 -- Gunnar Hjalmarsson <email address hidden> Tue, 31 Mar 2020 14:11:00 +0200

[Sebastien Bacher (seb128)]

@Gunnar, looks like the patch I backported earlier in the cycle
https://launchpad.net/ubuntu/+source/libxml2/2.9.4+dfsg1-8ubuntu3

But apparently doko synced from Debian overlooking we had a delta...

[Gunnar Hjalmarsson (gunnarhj)]

@Sebastien: Indeed, it's the very same thing. Then I spent several hours on an issue that was already spotted and fixed. :(

[Gunnar Hjalmarsson (gunnarhj)]

To the sponsor:

libxml2 2.9.10+dfsg-5, including the patch, is now available in Debian unstable. So please sync libxml2.

[Graham Inggs (ginggs)]

sync'd. please close this bug once libxml2 has been accepted
https://launchpad.net/ubuntu/focal/+queue?queue_state=1

[Launchpad Janitor (janitor)]

This bug was fixed in the package libxml2 - 2.9.10+dfsg-5

---------------
libxml2 (2.9.10+dfsg-5) unstable; urgency=medium

  * Team upload.

  [ Mattia Rizzolo ]
  * d/rules:
    + Drop --disable-silent-rules, already passed by dh_auto_configure.
    + Drop --parallel, now default with debhelper compat > 10.
    + Use dh_installdocs and dh_installexamples to install docs and examples.
    + Use dh_missing --fail-missing (and add the relevant d/not-installed).
    + Minimize indep build to build only the docs.
  * d/watch: fix an option to avoid a warning message.
  * d/control:
    + Move most of the build-deps to Build-Depends-Arch.
    + Use ${python:Depends} also for python-libxml2-dbg.
  * Add a lintian override for
    debian-rules-uses-supported-python-versions-without-python-all-build-depends

  [ Gunnar Hjalmarsson ]
  * d/p/python3-unicode-errors.patch:
    Fix segfault issue with itstool and py3. LP: #1869814

 -- Mattia Rizzolo <email address hidden> Fri, 10 Apr 2020 14:53:23 +0200