ltp-syscalls: msgstress03 / msgstress04 fails because systemd limits number of processes

[Thadeu Lima de Souza Cascardo]

As systemd limits the number of processes, this test will fail because it can't fork enough processes. That is limited to when the test is run after logging as user 1000, then running sudo. I guess that logging as root may not cause this to happen.

# ./testcases/bin/msgstress03
Fork failed (may be OK if under stress)
Fork failed (may be OK if under stress)
msgstress03 1 TFAIL : msgstress03.c:157: Fork failed (may be OK if under stress)
#

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1783881

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Dimitri John Ledkov (xnox)]

What about calling it like this:

$ sudo systemd-run ./testcases/bin/msgstress03

Does that make it pass correctly?

Which resource in particular is exhausted? and can it be toggled somehow using any of https://www.freedesktop.org/software/systemd/man/systemd.directives.html ?

[Thadeu Lima de Souza Cascardo (cascardo)]

The number of processes (in systemd, tasks).

The command below works for me.

Now, should we change the default on our test systems? Running under systemd-run does not look like a good option.

# systemd-run -p TasksMax=1000000 ./testcases/bin/msgstress03

[Dimitri John Ledkov (xnox)]

Hi,

I recommend you to change your test system.

For example, you can modify /etc/systemd/system.conf and change DefaultTasksMax there. But that is for the systemd started units...
Note that TasksMax these days can accept % values of kernel configured max tasks too, meaning i.e. one can set it to 100%..... The upstream default is 15% and we reverted that, meaning setting it to unlimited.

However something odd is going on.

I wonder if you are actually hitting UserTasksMax instead (which appears to be under-documented).

I wonder if setting UserTasksMax=1000000 in /etc/systemd/logind.conf in the [Login] section, restarting systemd-logind, creating a brand new user session (logout _all_ sessions, and relogin) would actually solve your problem?

ps. Also you can use a "drop-in" instead of modifying a config file, as all config files in systemd support .d `drop-ins` like so:

instead of modifying /etc/systemd/system.conf one can instead install files like these:

/{lib,etc,run}/systemd/system.conf.d/bump-tasks-max.conf

with like contents of
   [Manager]
   DefaultTasksMax=1000000

Depending on whether you want it to be packaged in a package, be a config file, or be a runtime adjustment.

[Sean Feole (sfeole)]

This failure occurs in the cloud, on amazon aws, it can be reproduced on c4/c5.large abd c5.metal

where it passes on c5n.xlarge / i3.metal m3.large m4.large.

I will try the suggested steps above on one of the affected flavor types and see if we can get to the bottom of this.

[Kelsey Steele (kelsey-steele)]

msgstress03 and msgstress04 is failing with similar behavior on Focal aws : 5.4.0-1026.26 : amd64

[Kelsey Steele (kelsey-steele)]

Updating bug to include msgstress04 failure.

msgstress04 - part of log from F aws : 5.4.0-1026.26 : amd64

12969. 09/23 17:30:15 DEBUG| utils:0153| [stdout] startup='Wed Sep 23 17:11:41 2020'
12970. 09/23 17:30:15 DEBUG| utils:0153| [stdout] msgstress04 0 TINFO : Found 32000 available message queues
12971. 09/23 17:30:15 DEBUG| utils:0153| [stdout] msgstress04 0 TINFO : Using upto 2097063 pids
12972. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9217
12973. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9237
12974. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9202
12975. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9251
12976. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9273
12977. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9277
12978. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9275
12979. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9276
12980. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9278
12981. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9279
12982. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9248
12983. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9262
12984. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9274
12985. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9310
12986. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9290
12987. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9314
12988. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9315
12989. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9316
12990. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9309
12991. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9325
12992. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9326
12993. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9298
12994. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9327
12995. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9308
12996. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the second child of child group 9324
12997. 09/23 17:30:15 DEBUG| utils:0153| [stdout] Fork failure in the first child of child group 9357
12998. 09/23 17:30:15 DEBUG| utils:0153| [stdout] msgstress04 1 TFAIL : msgstress04.c:204: Fork failed ...

[Po-Hsu Lin (cypressyew)]

Still visible 4.15.0-1059.65 - oracle
on instance VM.DenseIO2.8 only.

[Kelsey Steele (kelsey-steele)]

Found on Groovy/linux 5.8.0-31.33

[Marcelo Cerri (mhcerri)]

Still happening with linux-azure 5.8.0-1027.29 for cycle sru-20210315 but also for sru-20210222.

[Krzysztof Kozlowski (krzk)]

Found on focal/azure-5.8 5.8.0-1031.33~20.04.1

[Guilherme G. Piccoli (gpiccoli)]

Found on B-5.4/aws, cycle sru-20210531 .

[Krzysztof Kozlowski (krzk)]

Sent patch to upstream (LTP):
https://lists.linux.it/pipermail/ltp/2021-June/023346.html

[Dan Streetman (ddstreet)]

marking invalid for systemd as this doesn't seem like a systemd bug

[Guilherme G. Piccoli (gpiccoli)]

Observed in B/aws-5.4, cycle sru-20210621.

[Guilherme G. Piccoli (gpiccoli)]

Observed in B/aws (kernel 4.15), cycle sru-20210621.

[Guilherme G. Piccoli (gpiccoli)]

Observed in B/aws (kernel 4.15), cycle sru-20210621.