tail emits no output for sysfs files when using large page kernels

[dann frazier]

[Impact]
Ubuntu provides 64K page size kernels for ppc64el (always) and arm64 (optional -64k flavors). When booted on 64K kernels, tail emits no output when tailing a sysfs file. The difference in behavior can be a source for bugs in scripts that use tail, and general user confusion.

[Test Plan]
The upstream fix includes a test case that tails the /sys/kernel/profiling file, if it exists. That case would fail with an unfixed coreutils package as shown below:

= When booted on a 4K kernel =
ubuntu@gunyolk:~$ tail /sys/kernel/profiling
0

= When booted on a 64K kernel =
ubuntu@gunyolk:~$ tail /sys/kernel/profiling
ubuntu@gunyolk:~$

Since the upstream test cases are executed at build time, the existing tests and this new test will be used to regression test behavior. This should cover both 4K (!ppc64el) and 64K (ppc64el) cases. We should also do a manual verification on arm64 w/ the 64K kernel since that case is not covered by our builders.

[Where Problems Could Occur]
The biggest risk for a regression I see is due to the side-effect of the fix now allocating a dynamic buffer instead of the stack. An error in logic there could cause a crash or a memory leak in scenarios undetected during testing. I used valgrind when developing the fix to derisk the memory leak scenario.

[Launchpad Janitor (janitor)]

This bug was fixed in the package coreutils - 9.4-2ubuntu2

---------------
coreutils (9.4-2ubuntu2) noble; urgency=medium

  * d/p/tail-fix-tailing-sysfs-files-where-PAGE_SIZE-BUFSIZ.patch:
    Fix tailing of sysfs files on systems using a 64K page size.
    (LP: #2047450)

 -- dann frazier <email address hidden> Tue, 26 Dec 2023 11:25:09 -0700

[Steve Langasek (vorlon)]

An upload of coreutils to lunar-proposed has been rejected from the upload queue for the following reason: "lunar EOL in 9 days; no more SRUs".

[Timo Aaltonen (tjaalton)]

Hello dann, or anyone else affected,

Accepted coreutils into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/coreutils/9.1-1ubuntu2.23.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello dann, or anyone else affected,

Accepted coreutils into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/coreutils/8.32-4.1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[dann frazier (dannf)]

= mantic verification =
ubuntu@hinyari:~$ getconf PAGESIZE
65536
ubuntu@hinyari:~$ tail /sys/kernel/profiling
0
ubuntu@hinyari:~$ dpkg -l | grep coreutils
ii coreutils 9.1-1ubuntu2.23.10.1 arm64 GNU core utilities

[dann frazier (dannf)]

= jammy verification =
ubuntu@hinyari:~$ getconf PAGESIZE
65536
ubuntu@hinyari:~$ tail /sys/kernel/profiling
ubuntu@hinyari:~$ sudo apt install coreutils -t jammy-proposed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  coreutils
1 upgraded, 0 newly installed, 0 to remove and 42 not upgraded.
Need to get 1397 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-proposed/main arm64 coreutils arm64 8.32-4.1ubuntu1.1 [1397 kB]
Fetched 1397 kB in 1s (2153 kB/s)
(Reading database ... 83707 files and directories currently installed.)
Preparing to unpack .../coreutils_8.32-4.1ubuntu1.1_arm64.deb ...
Unpacking coreutils (8.32-4.1ubuntu1.1) over (8.32-4.1ubuntu1) ...
Setting up coreutils (8.32-4.1ubuntu1.1) ...
Processing triggers for install-info (6.8-4build1) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...
Scanning candidates...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
ubuntu@hinyari:~$ tail /sys/kernel/profiling
0

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted coreutils (8.32-4.1ubuntu1.1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

dgit/9.15 (amd64)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-gcp-5.19/5.19.0-1030.32~22.04.1 (arm64)
linux-gcp-6.2/6.2.0-1019.21~22.04.1 (arm64)
linux-hwe-5.19/5.19.0-50.50 (arm64)
linux-lowlatency/5.15.0-91.101 (arm64)
linux-lowlatency-hwe-5.19/5.19.0-1030.30 (arm64)
linux-lowlatency-hwe-6.5/6.5.0-14.14.1~22.04.1 (arm64)
linux-nvidia-6.5/6.5.0-1010.10 (arm64)
linux-nvidia-tegra/5.15.0-1020.20 (arm64)
linux-xilinx-zynqmp/5.15.0-1025.29 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#coreutils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted coreutils (9.1-1ubuntu2.23.10.1) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu15.1 (arm64, armhf, s390x)
linux-laptop/6.5.0-1009.12 (arm64)
linux-lowlatency/6.5.0-17.17.1 (arm64)
uqm/0.8.0+dfsg-2 (amd64, armhf, ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#coreutils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[dann frazier (dannf)]

I've examined all of the test failures, and all have root causes unrelated to this fix. I've cleared all from the excuses page except:

mantic/initramfs-tools/armhf: bug 2043579
jammy/linux-azure-5.19/arm64: the kernel build test timed out. retries keep hitting slow builders :(
jammy/linux-lowlatency-hwe-5.19/arm64: same
jammy/linux-nvidia-6.5: bug 2051941
jammy/linux-nvidia-tegra-igx/arm64: bug 2051942

Could the SRU team override those?

[Robie Basak (racb)]

I'm retrying these failing autopkgtests with migration-reference/0 since if they fail anyway they shouldn't hold things up.

In the meantime, could you please also verify that behaviour on 4K page size hasn't regressed? It looks like this was implied in the Test Plan but hasn't been verified. Thanks!

[dann frazier (dannf)]

> I'm retrying these failing autopkgtests with migration-reference/0 since if they fail anyway they shouldn't hold things up.

Oh, is that something I should do in cases like this? Would that automatically cause SRUs with failing tests to be unblocked?

> In the meantime, could you please also verify that behaviour on 4K page size hasn't regressed? It looks like this was implied in the Test Plan but hasn't been verified.

The patch includes a test case, which is shown to pass in the build logs.

From https://launchpadlibrarian.net/710103864/buildlog_ubuntu-mantic-arm64.coreutils_9.1-1ubuntu2.23.10.1_BUILDING.txt.gz :
  PASS: tests/tail-2/tail-sysfs.sh

From https://launchpadlibrarian.net/710103862/buildlog_ubuntu-jammy-arm64.coreutils_8.32-4.1ubuntu1.1_BUILDING.txt.gz :
  PASS: tests/tail-2/tail-sysfs.sh

I was considering that sufficient for 4K testing because the buildders run 4K kernels.

But it is trivial to do a manual test, and that would avoid having to trust that the test case itself didn't have any errors masking a failure, so:

root@mantic:~# getconf PAGESIZE
4096
root@mantic:~# dpkg -l coreutils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-====================-============-=================================
ii coreutils 9.1-1ubuntu2.23.10.1 arm64 GNU core utilities
root@mantic:~# tail /sys/kernel/profiling
0
root@mantic:~#
exit
ubuntu@bizzy:~$ lxc exec jammy -- /bin/bash
root@jammy:~# getconf PAGESIZE
4096
root@jammy:~# dpkg -l coreutils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=================================
ii coreutils 8.32-4.1ubuntu1.1 arm64 GNU core utilities
root@jammy:~# tail /sys/kernel/profiling
0
root@jammy:~#

[Andreas Hasenack (ahasenack)]

The migration-reference/0 runs passed, to the dep8 results are still not good. I retriggered the failing arm64 tests one more time.

[Andreas Hasenack (ahasenack)]

mantic is releseable, but I'll wait for the jammy results until the end of my shift. If they are still red, I can release just mantic.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for coreutils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package coreutils - 9.1-1ubuntu2.23.10.1

---------------
coreutils (9.1-1ubuntu2.23.10.1) mantic; urgency=medium

  * d/p/tail-fix-tailing-sysfs-files-where-PAGE_SIZE-BUFSIZ.patch:
    Fix tailing of sysfs files on systems using a 64K page size.
    (LP: #2047450)

 -- dann frazier <email address hidden> Sat, 06 Jan 2024 11:28:58 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package coreutils - 8.32-4.1ubuntu1.1

---------------
coreutils (8.32-4.1ubuntu1.1) jammy; urgency=medium

  * d/p/assure-new-macro-affirm.patch,
    d/p/tail-fix-tailing-sysfs-files-where-PAGE_SIZE-BUFSIZ.patch:
    Fix tailing of sysfs files on systems using a 64K page size.
    (LP: #2047450)

 -- dann frazier <email address hidden> Mon, 08 Jan 2024 07:56:36 -0700