pkgstripfiles: symlinking docs and optimizing pngs/trucating changelogs in parallel produces undeterministic results

[Balint Reczey]

[Impact]

 * PNG optimization and changelog truncation can happen in parallel with symlinking docs across packages. When checking for duplicates files the duplicate check may compare the processed files with non-processed ones and in that case they are found to be different despite the processed (or the unprocessed) files would be identical. The resulting binary packages would be still valid, but may be different for different architectures since the parallel build process may hit this race condition when processing different files making the arch:any packages not coinstallable on a multiarch-enabled system.
 * One manifestation of the problem can be seen in https://launchpadlibrarian.net/355140796/buildlog_ubuntu-bionic-i386.gpm_1.20.7-5_BUILDING.txt.gz where symlinking's file comparison occurs while truncating the changelog:
...
Searching for duplicated docs in dependency libgpm2...
pkgstripfiles: Truncating usr/share/doc/libgpm2/changelog.Debian.gz to topmost ten records

gzip: /<<PKGBUILDDIR>>/debian/libgpm-dev/../libgpm2/usr/share/doc/libgpm2/changelog.Debian.gz: unexpected end of file
cmp: EOF on /dev/fd/62 which is empty
...

 * The fix is protecting png optimization and trucating changelogs with the same lock as used to serialize doc symlinking thus those steps are serialized as well. The fix also skips optimizing PNGs and truncating changelogs of dbgsym files in pkgstripfiles.

[Test Case]

 * Build an affected package (like gpm) for i386 and amd64 with fixed pkgbinarymangler and observe the files in /usr/share/doc in the built packages to be identical and there should be no line like the following in the build log:
gzip: /<<PKGBUILDDIR>>/debian/libgpm-dev/../libgpm2/usr/share/doc/libgpm2/changelog.Debian.gz: unexpected end of file
 * Note that due to the non-deterministic nature of the failure several builds could be necessary to reproduce the original problem and also for verifying that the problem is fixed. You can increase the chances of observing the failure by building the packages with higher level of parallelism (-jX).

[Regression Potential]

 * Due to serialization the build times may increase for packages where the serialized steps ran in parallel originally. IMO there is not much we can do about that apart from trying to speed up the steps themselves.
 * Since no additional locks were introduced I believe the builds won't break or stall due to this change.
 * The fix also skips optimizing PNGs and truncating changelogs of dbgsym files, but PNG files are typically not present there and /usr/share/doc/<pkg>-dbgsym is symlinked to /usr/share/doc/<pkg> even with the fix thus the change of pkgstripfiles does not have an effect on the built dbgsym files' changelogs.

[Other Info]

 * There are some packages which need to be rebuilt with the updated pkgbinarymangler, collecting them is in progress.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 141

---------------
pkgbinarymangler (141) cosmic; urgency=medium

  * pkgstripfiles: Serialize PNG optimization and changelog trucation using
    the lock already used for serializing creating symlinks for identical
    documentation files. (LP: #1782094)

 -- Balint Reczey <email address hidden> Tue, 17 Jul 2018 10:03:52 +0200

[Robie Basak (racb)]

Hello Balint, or anyone else affected,

Accepted pkgbinarymangler into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/138.18.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Balint Reczey (rbalint)]

I have rebuilt gpm 200 times locally using not fixed pkgbinarymangler without observing the erratic behaviour where changelog is processed when checking identical files for symlinking.

OTOH it can be observed that thanks to moving PNG processing into the lock protected part PNG files are not optimized in dbgsym packages when using pkgbinarymangler/138.18.04.0:

$ grep pkgstr /home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T1* | grep 'pkgstripfiles.*dbgsym'
/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T10:34:29Z.build:pkgstripfiles: processing control file: debian/.debhelper/gpm/dbgsym-root/DEBIAN/control, package gpm-dbgsym, directory debian/.debhelper/gpm/dbgsym-root
/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T10:34:29Z.build:pkgstripfiles: processing control file: debian/.debhelper/libgpm2/dbgsym-root/DEBIAN/control, package libgpm2-dbgsym, directory debian/.debhelper/libgpm2/dbgsym-root
^^^ fixed

/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T11:00:21Z.build:pkgstripfiles: processing control file: debian/.debhelper/gpm/dbgsym-root/DEBIAN/control, package gpm-dbgsym, directory debian/.debhelper/gpm/dbgsym-root
/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T11:00:21Z.build:pkgstripfiles: Running PNG optimization (using 4 cpus) for package gpm-dbgsym ...
/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T11:00:21Z.build:pkgstripfiles: processing control file: debian/.debhelper/libgpm2/dbgsym-root/DEBIAN/control, package libgpm2-dbgsym, directory debian/.debhelper/libgpm2/dbgsym-root
/home/rbalint/ubuntu/logs/gpm_i386-2018-07-20T11:00:21Z.build:pkgstripfiles: Running PNG optimization (using 4 cpus) for package libgpm2-dbgsym ...
^^^ not fixed

Since the original problem is very hard to hit without instrumentation but I think it is clear when the root cause was and we can observe the side-effect of the fix I'm setting the bug to verified-done.

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for pkgbinarymangler has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 138.18.04.0

---------------
pkgbinarymangler (138.18.04.0) bionic; urgency=medium

  * pkgstripfiles: Serialize PNG optimization and changelog trucation using
    the lock already used for serializing creating symlinks for identical
    documentation files. (LP: #1782094)
  * pkgstripfiles: Skip symlinking identical documentation to not built
    packages, otherwise arch: any only builds may create symlinks which are
    not created when building arch: all and arch: any together. The mismatch
    between the symlinked and not symlinked files causes failure when trying
    to coinstall arch:any packages using multiarch.
    (LP: #1574351)

 -- Balint Reczey <email address hidden> Tue, 17 Jul 2018 16:23:13 +0200