glibc pwd/test-getpw test failures in autopkgtest

Bug #1869364 reported by Seth Forshee
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
glibc (Ubuntu)
High
Balint Reczey
Focal
Undecided
Unassigned
Groovy
High
Balint Reczey

Bug Description

[Impact]

pwd/test-getpw test fails in autopkgtest
Fixing this makes the arm64 and armhf autopkgtests usable again for the lifetime of the 20.04 LTS release.

[Test Case]

Observe autopkgetest tests passing, including pwd/test-getpw being marked as XFAIL.

[Regression Potential]

The fix is ignoring a test failure due to the problems explained in the bug comments. The fix does not affect run-time behaviour.

[Original Bug Text]

I started seeing this in the testing for 5.4.0-20. Only happening on arm64/armhf.

----------
FAIL: pwd/tst-getpw
original exit status 223
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
FAIL: Did not find even one invalid uid.
----------

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/g/glibc/20200326_165114_2fcf9@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/armhf/g/glibc/20200326_173356_3378d@/log.gz

I've unsuccessfully tried to reproduce this on an arm64 host (by running this test case standalone, not the full test suite), so possibly there's something about the autopkgtest environment. This also happens with the 5.4.0-18 from focal-release, so it does not appear to be a regression in the kernel.

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/arm64/g/glibc/20200326_224201_5fd33@/log.gz

Seth Forshee (sforshee)
summary: - glibc pwd/test-pwd test failures in autopkgtest
+ glibc pwd/test-getpw test failures in autopkgtest
Revision history for this message
Matthias Klose (doko) wrote :

still seen in 2.31-0ubuntu7

Revision history for this message
Florian Weimer (fweimer) wrote :

I believe this is related to the kind of NSS modules which are active on the system. I believe it's related how the way errno is (not) set. I'm not sure if this is a bug in the NSS framework, in the NSS module, or simply a case of incorrect test expectations.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glibc (Ubuntu):
status: New → Confirmed
Balint Reczey (rbalint)
tags: added: update-excuse
tags: added: rls-gg-incoming
Changed in glibc (Ubuntu):
importance: Undecided → High
Changed in glibc (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote :

Sorry, this was closed incorrectly based on bad information from me. The autopkgtests are still failing.

Changed in glibc (Ubuntu):
status: Fix Released → Triaged
tags: removed: rls-ff-incoming rls-gg-incoming
tags: added: id-5ec6a6fd770fb949059e2e9a
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Shouldn't the test set errno back to 0 after triggering the EINVAL?

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Well maybe it should but that doesn't help the test :/

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

So I managed to reproduce the test failure in an arm64 vm, and narrowed it down to this:

ubuntu@juju-b11c42-ubuntu-24:/tmp/autopkgtest.0yUqH9/build.EJr/src/build-tree/arm64-libc/pwd$ ./tst-getpw
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
PASS: Found an invalid uid.
ubuntu@juju-b11c42-ubuntu-24:/tmp/autopkgtest.0yUqH9/build.EJr/src/build-tree/arm64-libc/pwd$ /lib/ld-linux-aarch64.so.1 ./tst-getpw
PASS: NULL buffer returns -1 and sets errno to EINVAL.
PASS: Read a password line given a uid.
FAIL: Did not find even one invalid uid.

which doesn't make a lot of sense but is somewhere to start...

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Was the ./tst-getpw just compiled? and should it be using linker from build directory? what is the interpret set on the ./tst-getpw file?

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Yes, and it doesn't matter whether you use the interpreter from the build directory or the default one (the binary has PF_INTERP set to /lib/ld-linux-aarch64.so.1 but the testsuite invokes it with the one from the build tree).

Some gdb time later, it transpires that this is systemd's fault :) (at least partially)

I'm a bit hazy on NSS in general, but valid uids present in /etc/passwd are (I assume) found by the "compat" nss module. Ones that are not found there are sent off to the libnss-systemd.so.

In there, _nss_systemd_getpwuid_r calls userdb_getpwuid which calls userdb_nss_compat_is_enabled which calls userdb_thread_sockaddr which calls getauxval(AT_RANDOM) which for some reason in this situation _fails_. userdb_thread_sockaddr then returns -EIO which leads to userdb_getpwuid setting errnop to EIO and returning NSS_STATUS_UNAVAIL which leads to _nss_systemd_getpwuid_r setting its errnop to EIO and that's what the test program sees as errno.

This leads me to this experiment:

ubuntu@juju-b11c42-ubuntu-24:~$ cat getauxval.c
#include <sys/auxv.h>
#include <stdio.h>

int main(int argc, char** argv) {
        printf("%lu\n", getauxval(AT_RANDOM));
}
ubuntu@juju-b11c42-ubuntu-24:~$ gcc -o getauxval getauxval.c
ubuntu@juju-b11c42-ubuntu-24:~$ ./getauxval
281474825715272
ubuntu@juju-b11c42-ubuntu-24:~$ /lib/ld-linux-aarch64.so.1 ./getauxval
0

which seems like a bug (it doesn't happen on amd64 or s390x).

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

This was reported upstream in 2018 https://sourceware.org/bugzilla/show_bug.cgi?id=23293, no activity though

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

On thinking about this, the failing test isn't reporting a regression so I'll go ahead and mark the test as xfail for now.

Changed in glibc (Ubuntu Groovy):
status: Triaged → In Progress
tags: added: id-5ebae3f15f6fda125f376d66
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

attaching the debdiff so it can be included in the next upload

tags: added: patch
tags: added: id-5f117d345c4abc523e9b7b67
Balint Reczey (rbalint)
Changed in glibc (Ubuntu Groovy):
assignee: nobody → Balint Reczey (rbalint)
Balint Reczey (rbalint)
description: updated
Revision history for this message
Balint Reczey (rbalint) wrote :

This is happening on armhf as well so I extend the patch to cover armhf, too.

root@gg-libc-test:~# /lib/ld-linux-armhf.so.3 ./getauxval
0

Balint Reczey (rbalint)
description: updated
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Seth, or anyone else affected,

Accepted glibc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in glibc (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glibc - 2.31-0ubuntu11

---------------
glibc (2.31-0ubuntu11) groovy; urgency=medium

  [ Michael Hudson-Doyle ]
  * Mark tst-getpw as XFAIL on arm64. (LP: #1869364)

  [ Balint Reczey ]
  * debian/control: Add Vcs-* pointing to Ubuntu packaging repository
  * debian/gbp.conf: Add initial configuration
  * debian/debhelper.in/libc.preinst: Fix setting LDCONFIG_NOTRIGGER
  * debian/control.in/main: Add Vcs-* pointing to Ubuntu packaging repository
  * Fall back to calling nanosleep syscall when __clock_nanosleep returns EINVAL
    (LP: #1871240)
  * debian/testsuite-xfail-debian.mk: XFAIL stdlib/tst-strtod-round on riscv64
  * debian/testsuite-xfail-debian.mk: XFAIL tst-getpw on armhf, too
  * debian/watch: Use HTTPS and download xz-compressed tarball
  * debian/watch: Use upstream's signing key to verify the tarball
  * XFAIL stdlib/tst-getrandom (LP: #1891403)

  [ Dimitri John Ledkov ]
  * debian/patches/powerpc: Cherrypick upstream patches to support POWER10
    optimized library loading. (LP: #1887989)

  [ Aurelien Jarno ]
  * debian/patches/any/submitted-selinux-deprecations.diff: proposed patch to
    ignore the selinux deprecations introduced in libselinux (>= 3.1), fixing
    an FTBFS. (Closes: #965941)

 -- Balint Reczey <email address hidden> Thu, 27 Aug 2020 23:36:39 +0200

Changed in glibc (Ubuntu Groovy):
status: In Progress → Fix Released
Revision history for this message
Balint Reczey (rbalint) wrote :
tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (glibc/2.31-0ubuntu9.1)

All autopkgtests for the newly accepted glibc (2.31-0ubuntu9.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-blackbox-exporter/0.13.0+ds-2 (armhf, amd64, ppc64el, s390x, arm64)
prometheus-pushgateway/1.0.0+ds-1 (armhf, amd64, ppc64el, s390x, arm64)
systemd/245.4-4ubuntu3.2 (s390x, amd64, ppc64el)
gfs2-utils/unknown (amd64)
hugo/0.68.3-1 (armhf, amd64, ppc64el, s390x, arm64)
grubzfs-testsuite/0.4.10 (amd64)
glibc/2.31-0ubuntu9.1 (armhf)
badger/2.0.1-3 (armhf, amd64, ppc64el, s390x, arm64)
resource-agents/1:4.5.0-2ubuntu2 (armhf)
etcd/3.2.26+dfsg-6 (amd64, ppc64el)
postgresql-multicorn/1.3.4-31-g9ff7875-3 (armhf, amd64, ppc64el, s390x)
gfs2-utils/3.2.0-3 (ppc64el, s390x, arm64)
scipy/1.3.3-3build1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glibc - 2.31-0ubuntu9.1

---------------
glibc (2.31-0ubuntu9.1) focal; urgency=medium

  [ Michael Hudson-Doyle ]
  * Mark tst-getpw as XFAIL on arm64. (LP: #1869364)

  [ Matthias Klose ]
  * Copy the fully conditionalized x86 variant for math-vector-fortran.h
    to /usr/include/finclude. On all architectures. (LP: #1879092)

  [ Balint Reczey ]
  * debian/gbp.conf: Add initial configuration
  * debian/control.in/main: Add Vcs-* pointing to Ubuntu packaging repository
  * debian/debhelper.in/libc.preinst: Fix setting LDCONFIG_NOTRIGGER
    (LP: #1889190)
  * Fall back to calling nanosleep syscall when __clock_nanosleep returns
    EINVAL due to CLOCK_REALTIME not being supported (LP: #1871129)
  * debian/testsuite-xfail-debian.mk: XFAIL tst-getpw on armhf, too
    (LP: #1869364)
  * XFAIL stdlib/tst-getrandom (LP: #1891403)

  [ Dimitri John Ledkov ]
  * debian/patches/powerpc: Cherrypick upstream patches to support POWER10
    optimized library loading. LP: #1887989

 -- Balint Reczey <email address hidden> Mon, 17 Aug 2020 22:02:52 +0200

Changed in glibc (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for glibc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.