pgrep reports error "cannot allocate" when run without stack limit

[Sven Hartrumpf]

[Impact]

  - Users who have ulimit set high would see either slow
    or failed pgrep and pkill commands
  - Users who have ulimit set to unlimited would see
    failed pgrep and pkill commands
  - This bug occurs because the behavior of sysconf(_SC_ARG_MAX)
    changed with a newer version of the kernel.

[Test Case]

  - set the ulimit to unlimited by running `ulimit -S -s unlimited`
  - run `pgrep bash` to see that the "cannot allocate" error is
     printed and the command has failed.

[Where Problems Could Occur]

  - We have set upper and lower limits on the size of the malloc, but
    if further kernel versions break the call to sysconf in
    unexpected ways we could still see problems.

[Original Description]

If you have no stack limit (ulimit -S -s unlimited), any pgrep call will fail with an error:

> pgrep vim
pgrep: cannot allocate 4611686018427387903 bytes

If you have a high stack limit (e.g. ulimit -S -s 500000), pgrep is very slow:

> time pgrep vim
2196
real 8.48s user 8.40s syst 0.07s busy 99% rmem 253444

The relevant upstream bug report could be: https://gitlab.com/procps-ng/procps/-/issues/152
Archlinux bug report: https://bugs.archlinux.org/task/66093

procps:
  Installed: 2:3.3.16-1ubuntu2
  500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Sven Hartrumpf (hartrumpf)]

For the unlimited case, there is also Debian bug #955697 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955697
Those bug reports do not mention the massive slowdown linear in stack limit.

[Sven Hartrumpf (hartrumpf)]

The problem is the same for other popular tools like pkill from the same package.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sven Hartrumpf (hartrumpf)]

Archlinux fixed this bug today, now in testing stage. Any news for Ubuntu?

[Sven Hartrumpf (hartrumpf)]

I installed procps-ng 3.3.16.49-ae4a myself on all our Ubuntu 20.04 computers. This fixed both problems, slow pgrep / pkill and crashing pgrep / pkill.

[Sven Hartrumpf (hartrumpf)]

The Arch Linux fix appeared in Manjaro. I can confirm that it works on Manjaro machines.

[Sven Hartrumpf (hartrumpf)]

Even worse for Ubuntu 20.10 (procps 2:3.3.16-5ubuntu2). With a stack limit of 500000, almost 11 seconds for a simple pgrep; without stacklimit, immediate crash.

[Vasilis (vasilis-vlachoudis)]

It appeared today, after rebooting with the latest kernel, however I have not updated procps since some time now and it was working before

[Sven Hartrumpf (hartrumpf)]

Can the bug fix be applied for Ubuntu 21.04? Please.

On slow machines like a Raspberry 4 with stack limit 500000 (Ubuntu 20.10), a single pgrep call takes 17 seconds!

[Robie Basak (racb)]

Patch available at https://gitlab.com/procps-ng/procps/-/commit/bb96fc42956c9ed926a1b958ab715f8b4a663dec

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Luke Yeager (luke-yeager)]

FWIW, I was able to apply bb96fc4295 cleanly to the current 20.04 package (2:3.3.16-1ubuntu2) and it works for me.

apt-get -y source procps && cd procps-*
curl -fSsL -o debian/patches/sc_arg_max https://gitlab.com/procps-ng/procps/-/commit/bb96fc4295.patch
echo sc_arg_max >>debian/patches/series
dch --local myorg 'Apply patch: "pgrep: check sanity of SC_ARG_MAX"'
debuild -us -uc -G -i -tc

[William Wilson (jawn-smith)]

Attached is a patch to fix the issue in hirsute. Assuming this is approved, I have also prepared the SRUs for focal and groovy.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Hirsute debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Brian Murray (brian-murray)]

I've sponsored this for hirsute but changed the package version to not include the release number as this isn't an SRU for hirsute.

[Launchpad Janitor (janitor)]

This bug was fixed in the package procps - 2:3.3.16-5ubuntu3

---------------
procps (2:3.3.16-5ubuntu3) hirsute; urgency=medium

  * Address slowness and crashes with large or unlimited stack limits
    (LP: #1874824)

 -- William 'jawn-smith' Wilson <email address hidden> Wed, 24 Mar 2021 10:09:08 -0500

[Balint Reczey (rbalint)]

I've sponsored the SRUs.

[Robie Basak (racb)]

> The problem is the same for other popular tools like pkill from the same package.

The SRU uploads fix pgrep, but not pkill. Isn't pkill also affected? Rather than fixing that in a separate SRU causing users to have to update twice, should we be fixing both at the same time? Or are you intending to never fix pkill in an SRU? Are there any other commands affected by the same issue?

[William Wilson (jawn-smith)]

Robie,

pgrep and pkill are sharing the same code. Essentially pkill runs pgrep and then issues a kill signal to the resulting pid. Therefore this fix applies to both pkill and pgrep. Additionally, the source of the problem comes from the behavior of `sysconf(_SC_ARG_MAX)` changing in newer kernels. This function call only exists in one place in the procps codebase, so no other commands are affected by the same issue.

[Robie Basak (racb)]

Ah, sorry, my mistake. I didn't think that pgrep.c would be used for a different command. Now I see that pkill is a symlink to pgrep in /usr/bin.

[Robie Basak (racb)]

Hello Sven, or anyone else affected,

Accepted procps into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/procps/2:3.3.16-5ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Sven, or anyone else affected,

Accepted procps into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/procps/2:3.3.16-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted procps (2:3.3.16-1ubuntu2.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ranger/1.9.3-1build1 (s390x)
xpra/3.0.6+dfsg1-1build1 (amd64)
reprotest/0.7.14 (ppc64el, amd64)
s3ql/3.3.2+dfsg-1ubuntu1 (armhf)
apport/2.20.11-0ubuntu27.16 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#procps

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Sven Hartrumpf (hartrumpf)]

I tested procps/groovy-proposed,now 2:3.3.16-5ubuntu2.1 amd64. I have repeated pgrep and pkill commands that caused the bug before. They do work with the proposed package.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted procps (2:3.3.16-5ubuntu2.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

ranger/1.9.3-2 (s390x)
reprotest/0.7.15 (s390x)
postorius/1.3.2-2.1 (arm64)
nodejs/12.18.2~dfsg-1ubuntu2 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#procps

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[William Wilson (jawn-smith)]

The verification passed for focal:

jawn-smith@focal-vm:~$ apt-cache policy procps
procps:
  Installed: 2:3.3.16-1ubuntu2
  Candidate: 2:3.3.16-1ubuntu2.1
  Version table:
     2:3.3.16-1ubuntu2.1 500
        500 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
 *** 2:3.3.16-1ubuntu2 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status
jawn-smith@focal-vm:~$ ulimit -S -s unlimited
jawn-smith@focal-vm:~$ pgrep bash
pgrep: cannot allocate 4611686018427387903 bytes
jawn-smith@focal-vm:~$ sudo apt install procps
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  procps
1 upgraded, 0 newly installed, 0 to remove and 64 not upgraded.
Need to get 233 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 procps amd64 2:3.3.16-1ubuntu2.1 [233 kB]
Fetched 233 kB in 0s (613 kB/s)
(Reading database ... 184500 files and directories currently installed.)
Preparing to unpack .../procps_2%3a3.3.16-1ubuntu2.1_amd64.deb ...
Unpacking procps (2:3.3.16-1ubuntu2.1) over (2:3.3.16-1ubuntu2) ...
Setting up procps (2:3.3.16-1ubuntu2.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.6) ...
jawn-smith@focal-vm:~$ ulimit -S -s unlimited
jawn-smith@focal-vm:~$ pgrep bash
1744
jawn-smith@focal-vm:~$

[Launchpad Janitor (janitor)]

This bug was fixed in the package procps - 2:3.3.16-5ubuntu2.1

---------------
procps (2:3.3.16-5ubuntu2.1) groovy; urgency=medium

  * Address slowness and crashes with large or unlimited stack limits (LP: #1874824)

 -- William 'jawn-smith' Wilson <email address hidden> Wed, 24 Mar 2021 10:09:08 -0500

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for procps has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package procps - 2:3.3.16-1ubuntu2.1

---------------
procps (2:3.3.16-1ubuntu2.1) focal; urgency=medium

  * Address slowness and crashes with large or unlimited stack limits (LP: #1874824)

 -- William 'jawn-smith' Wilson <email address hidden> Wed, 24 Mar 2021 13:51:32 -0500