Broken on ppc64el (toolchain bug?)

[Chris Halse Rogers]

umockdev appears to be broken on ppc64el in impish. Running it on one of Mir's umockdev-using tests results in:

(impish-ppc64el)root@juju-deb017-porterbox-1:/build/mir-Xn1VqE/umockdev# umockdev-run ../mir-2.4.1/build-ppc64el/bin/mir_umock_unit_tests
MIR_CLIENT_PLATFORM_PATH=../mir-2.4.1/build-ppc64el/bin/../lib/client-modules/
MIR_SERVER_PLATFORM_PATH=../mir-2.4.1/build-ppc64el/bin/../lib/server-modules/
LD_LIBRARY_PATH=../mir-2.4.1/build-ppc64el/bin/../lib
exec=../mir-2.4.1/build-ppc64el/bin/mir_umock_unit_tests.bin
*** stack smashing detected ***: terminated
umockdev-run: unable to propagate signal 6 to child 15833: No such process

(You can also see this in the Mir 2.4.1-0ubuntu1 build log: https://launchpadlibrarian.net/546972958/buildlog_ubuntu-impish-ppc64el.mir_2.4.1-0ubuntu1_BUILDING.txt.gz )

Installing umockdev 0.15.4-1 and libumockdev0 0.15.4-1 from hirsute results in those tests passing.

Strangely, rebuilding umockdev 0.15.4-1 in a hirsute sbuild environment results in packages that do *not* pass those tests, suggesting a toolchain change might be responsible.

Unfortunately, I've tried rebuilding umockdev with gcc-9, gcc-11, and vala 0.48.12-1 in Impish and none of these appear to work.

ProblemType: Bug
DistroRelease: Ubuntu 21.10
Package: umockdev 0.16.1-1
ProcVersionSignature: Ubuntu 5.11.0-20.21+21.10.1-generic 5.11.21
Uname: Linux 5.11.0-20-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu67
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Thu Jul 8 16:04:15 2021
InstallationDate: Installed on 2021-06-26 (11 days ago)
InstallationMedia: Ubuntu 21.10.0 2021.05.28 amd64 "bcachefs" (20210622)
SourcePackage: umockdev
UpgradeStatus: No upgrade log present (probably fresh install)

[Martin Pitt (pitti)]

Dang, we already found a ppc64el SIGBUS issue in 0.16.0, which got fixed in https://github.com/martinpitt/umockdev/commit/277c80243a . But this is reported against 0.16.1 already.

There is a tiny chance that https://github.com/martinpitt/umockdev/commit/264cabbb will magically fix this, but otherwise this needs some investigation. I.e. not known upstream yet.

[Chris Halse Rogers (raof)]

Seb has rebuilt bolt in impish, which uses umockdev for some of its tests, and that apparently works ( https://matrix.to/#/!cVrEyzKyyNYOVfcQsb:libera.chat/$az_4Y_uIkZ02vMS6WjfQlkTvOH_ikJQTCkqRsAcOBfY?via=libera.chat&via=matrix.org&via=cooperteam.net )

The stack-smashing backtrace implicates LTTNG, so maybe there's some strange interaction there?

I'm not sure this is an upstream bug, as rebuilding 0.15.4 makes the crash appear.

[Chris Halse Rogers (raof)]

Yeah, whatever change in dependencies causes this it goes back to hirsute release.

Running mir 2.3.3's tests against 0.15.4-1 from the hirsute archive works; running those tests against 0.15.4-1 rebuilt in a hirsute chroot fails.

[Chris Halse Rogers (raof)]

Next possible culprit: binutils

[Chris Halse Rogers (raof)]

Ok: A list of things that it appears *not* to be:
1) binutils, valac, gcc versions. A umockdev build with the versions that hirsute's 0.15.4-1 built with.
2) The "debugedit: debian/umockdev/usr/bin/umockdev-record: Unknown DWARF DW_FORM_0x1f20" messages; building an unstripped umockdev doesn't emit this message (as it's no longer invoked by dh_strip), but the built package still results in the Mir tests failing as above.

[Michael Hudson-Doyle (mwhudson)]

So I don't have the faintest idea what caused this to start failing but the issue here is in mir:

    int (*real_open)(char const *path, int flags, mode_t mode);
    *(void **)(&real_open) = dlsym(RTLD_NEXT, "open");

    return (*real_open)(path, flags, mode);

The declaration for real_open here does not match that used by the open() function call in umockdev, which uses va_args. This means that the stack frame gcc creates for mir's wrapper does not have space for a parameter save area which the code gcc generates for umockdev assumes it has, and so the stack gets stomped on. So if the prototype in mir is fixed here (also the one for open64) I bet things will start working again.

[Martin Pitt (pitti)]

Indeed the open(2) manpage is misleading in that regard. The actual definition in fcntl.h is like this:

    extern int open (const char *__file, int __oflag, ...) __nonnull ((1));

(with a few variants, but they all use varargs). So I did the same in umockdev for full header compatibility.

[Simon Chopin (schopin)]

Blocks migration to -updates as the fixed mir package FTBFSe.

[Michael Hudson-Doyle (mwhudson)]

Martin, right I thought I'd said that in my comment too about the prototype in the header but apparently I only thought it really loudly, or something :)

[Michael Hudson-Doyle (mwhudson)]

Simon, no fix has been uploaded yet?

[Simon Chopin (schopin)]

Oh, sorry, I know see how my sentence is ambiguous. I meant that the failing autopkgtests are fixed in the latest version of mir uploaded to -proposed, but that version fails to build because of this bug. So the mentioned fix is *not* for this bug.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mir - 2.4.1-0ubuntu2

---------------
mir (2.4.1-0ubuntu2) impish; urgency=medium

  * debian/patches/fix-UB-in-open-interposer.patch:
    - Cherry-pick upstream patches fixing UB in the test-suite, resulting in a
      FTBFS on ppc64el (LP: #1934995)
  * debian/patches/work-around-gcc-maybe-uninitialised-bug.patch:
    - Cherry-pick upstream patches working around an incorrect warning
      caused by a gcc bug exposed with the previous patch.

 -- Christopher James Halse Rogers <email address hidden> Wed, 28 Jul 2021 11:01:36 +1000