glibc pwd/test-getpw test failures in autopkgtest

[Seth Forshee]

[Impact]

pwd/test-getpw test fails in autopkgtest
Fixing this makes the arm64 and armhf autopkgtests usable again for the lifetime of the 20.04 LTS release.

[Test Case]

Observe autopkgetest tests passing, including pwd/test-getpw being marked as XFAIL.

[Regression Potential]

The fix is ignoring a test failure due to the problems explained in the bug comments. The fix does not affect run-time behaviour.

[Original Bug Text]

I started seeing this in the testing for 5.4.0-20. Only happening on arm64/armhf.

----------
FAIL: pwd/tst-getpw
original exit status 223
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
FAIL: Did not find even one invalid uid.
----------

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/g/glibc/20200326_165114_2fcf9@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/armhf/g/glibc/20200326_173356_3378d@/log.gz

I've unsuccessfully tried to reproduce this on an arm64 host (by running this test case standalone, not the full test suite), so possibly there's something about the autopkgtest environment. This also happens with the 5.4.0-18 from focal-release, so it does not appear to be a regression in the kernel.

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/g/glibc/20200326_224201_5fd33@/log.gz

[Matthias Klose (doko)]

still seen in 2.31-0ubuntu7

[Florian Weimer (fweimer)]

I believe this is related to the kind of NSS modules which are active on the system. I believe it's related how the way errno is (not) set. I'm not sure if this is a bug in the NSS framework, in the NSS module, or simply a case of incorrect test expectations.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Sorry, this was closed incorrectly based on bad information from me. The autopkgtests are still failing.

[Michael Hudson-Doyle (mwhudson)]

Shouldn't the test set errno back to 0 after triggering the EINVAL?

[Michael Hudson-Doyle (mwhudson)]

Well maybe it should but that doesn't help the test :/

[Michael Hudson-Doyle (mwhudson)]

So I managed to reproduce the test failure in an arm64 vm, and narrowed it down to this:

ubuntu@juju-b11c42-ubuntu-24:/tmp/autopkgtest.0yUqH9/build.EJr/src/build-tree/arm64-libc/pwd$ ./tst-getpw
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
PASS: Found an invalid uid.
ubuntu@juju-b11c42-ubuntu-24:/tmp/autopkgtest.0yUqH9/build.EJr/src/build-tree/arm64-libc/pwd$ /lib/ld-linux-aarch64.so.1 ./tst-getpw
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
FAIL: Did not find even one invalid uid.

which doesn't make a lot of sense but is somewhere to start...

[Dimitri John Ledkov (xnox)]

Was the ./tst-getpw just compiled? and should it be using linker from build directory? what is the interpret set on the ./tst-getpw file?

[Michael Hudson-Doyle (mwhudson)]

Yes, and it doesn't matter whether you use the interpreter from the build directory or the default one (the binary has PF_INTERP set to /lib/ld-linux-aarch64.so.1 but the testsuite invokes it with the one from the build tree).

Some gdb time later, it transpires that this is systemd's fault :) (at least partially)

I'm a bit hazy on NSS in general, but valid uids present in /etc/passwd are (I assume) found by the "compat" nss module. Ones that are not found there are sent off to the libnss-systemd.so.

In there, _nss_systemd_getpwuid_r calls userdb_getpwuid which calls userdb_nss_compat_is_enabled which calls userdb_thread_sockaddr which calls getauxval(AT_RANDOM) which for some reason in this situation _fails_. userdb_thread_sockaddr then returns -EIO which leads to userdb_getpwuid setting errnop to EIO and returning NSS_STATUS_UNAVAIL which leads to _nss_systemd_getpwuid_r setting its errnop to EIO and that's what the test program sees as errno.

This leads me to this experiment:

ubuntu@juju-b11c42-ubuntu-24:~$ cat getauxval.c
#include <sys/auxv.h>
#include <stdio.h>

int main(int argc, char** argv) {
        printf("%lu\n", getauxval(AT_RANDOM));
}
ubuntu@juju-b11c42-ubuntu-24:~$ gcc -o getauxval getauxval.c
ubuntu@juju-b11c42-ubuntu-24:~$ ./getauxval
281474825715272
ubuntu@juju-b11c42-ubuntu-24:~$ /lib/ld-linux-aarch64.so.1 ./getauxval
0

which seems like a bug (it doesn't happen on amd64 or s390x).

[Michael Hudson-Doyle (mwhudson)]

This was reported upstream in 2018 https://sourceware.org/bugzilla/show_bug.cgi?id=23293, no activity though

[Michael Hudson-Doyle (mwhudson)]

On thinking about this, the failing test isn't reporting a regression so I'll go ahead and mark the test as xfail for now.

[Michael Hudson-Doyle (mwhudson)]

attaching the debdiff so it can be included in the next upload

[Balint Reczey (rbalint)]

This is happening on armhf as well so I extend the patch to cover armhf, too.

root@gg-libc-test:~# /lib/ld-linux-armhf.so.3 ./getauxval
0

[Robie Basak (racb)]

Hello Seth, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.31-0ubuntu11

---------------
glibc (2.31-0ubuntu11) groovy; urgency=medium

  [ Michael Hudson-Doyle ]
  * Mark tst-getpw as XFAIL on arm64. (LP: #1869364)

  [ Balint Reczey ]
  * debian/control: Add Vcs-* pointing to Ubuntu packaging repository
  * debian/gbp.conf: Add initial configuration
  * debian/debhelper.in/libc.preinst: Fix setting LDCONFIG_NOTRIGGER
  * debian/control.in/main: Add Vcs-* pointing to Ubuntu packaging repository
  * Fall back to calling nanosleep syscall when __clock_nanosleep returns EINVAL
    (LP: #1871240)
  * debian/testsuite-xfail-debian.mk: XFAIL stdlib/tst-strtod-round on riscv64
  * debian/testsuite-xfail-debian.mk: XFAIL tst-getpw on armhf, too
  * debian/watch: Use HTTPS and download xz-compressed tarball
  * debian/watch: Use upstream's signing key to verify the tarball
  * XFAIL stdlib/tst-getrandom (LP: #1891403)

  [ Dimitri John Ledkov ]
  * debian/patches/powerpc: Cherrypick upstream patches to support POWER10
    optimized library loading. (LP: #1887989)

  [ Aurelien Jarno ]
  * debian/patches/any/submitted-selinux-deprecations.diff: proposed patch to
    ignore the selinux deprecations introduced in libselinux (>= 3.1), fixing
    an FTBFS. (Closes: #965941)

 -- Balint Reczey <email address hidden> Thu, 27 Aug 2020 23:36:39 +0200

[Balint Reczey (rbalint)]

Tested 2.31-0ubuntu9.1 on Focal:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/g/glibc/20200903_074430_f4967@/log.gz
...
XFAIL: pwd/tst-getpw
XFAIL: pwd/tst-getpw
...

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/armhf/g/glibc/20200908_023515_757dc@/log.gz
...
XFAIL: pwd/tst-getpw
XFAIL: pwd/tst-getpw
...

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-blackbox-exporter/0.13.0+ds-2 (armhf, amd64, ppc64el, s390x, arm64)
prometheus-pushgateway/1.0.0+ds-1 (armhf, amd64, ppc64el, s390x, arm64)
systemd/245.4-4ubuntu3.2 (s390x, amd64, ppc64el)
gfs2-utils/unknown (amd64)
hugo/0.68.3-1 (armhf, amd64, ppc64el, s390x, arm64)
grubzfs-testsuite/0.4.10 (amd64)
glibc/2.31-0ubuntu9.1 (armhf)
badger/2.0.1-3 (armhf, amd64, ppc64el, s390x, arm64)
resource-agents/1:4.5.0-2ubuntu2 (armhf)
etcd/3.2.26+dfsg-6 (amd64, ppc64el)
postgresql-multicorn/1.3.4-31-g9ff7875-3 (armhf, amd64, ppc64el, s390x)
gfs2-utils/3.2.0-3 (ppc64el, s390x, arm64)
scipy/1.3.3-3build1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.31-0ubuntu9.1

---------------
glibc (2.31-0ubuntu9.1) focal; urgency=medium

  [ Michael Hudson-Doyle ]
  * Mark tst-getpw as XFAIL on arm64. (LP: #1869364)

  [ Matthias Klose ]
  * Copy the fully conditionalized x86 variant for math-vector-fortran.h
    to /usr/include/finclude. On all architectures. (LP: #1879092)

  [ Balint Reczey ]
  * debian/gbp.conf: Add initial configuration
  * debian/control.in/main: Add Vcs-* pointing to Ubuntu packaging repository
  * debian/debhelper.in/libc.preinst: Fix setting LDCONFIG_NOTRIGGER
    (LP: #1889190)
  * Fall back to calling nanosleep syscall when __clock_nanosleep returns
    EINVAL due to CLOCK_REALTIME not being supported (LP: #1871129)
  * debian/testsuite-xfail-debian.mk: XFAIL tst-getpw on armhf, too
    (LP: #1869364)
  * XFAIL stdlib/tst-getrandom (LP: #1891403)

  [ Dimitri John Ledkov ]
  * debian/patches/powerpc: Cherrypick upstream patches to support POWER10
    optimized library loading. LP: #1887989

 -- Balint Reczey <email address hidden> Mon, 17 Aug 2020 22:02:52 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for glibc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.