ubuntu_ltp_controllers tests failing on Impish ($cgroup_name already mounted or mount point busy)

[Luke Nowakowski-Krijger]

Almost half of the ubuntu_ltp_controllers tests (142 out of 198) are failing due to the general pattern 'cgroup_name already mounted or mount point busy' causing the tests to fail.

e.g.
mount: /dev/cgroup: ltp_cgroup already mounted or mount point busy.
cgroup_fj_function2_memory 1 TBROK: mount -t cgroup -o memory ltp_cgroup /dev/cgroup failed

From investigation it seems there could be an issue with the transition to cgroup-v2. There have been rumors on the ltp mailing list that one of these days the tests could break due to the transition. Switching to cgroup-v2, likely due to a systemd update, could cause these tests to break due to different mount and cgroup hierarchy semantics.

I could only reproduce a subset of the new failures we are seeing, but after setting systemd.unified_cgroup_hierarchy=0 on the kernel command line which sets cgroup back to v1, a lot of the failures I could produce went away.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1949532

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Luke Nowakowski-Krijger (lukenow)]

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1942113

Here is a discussion about recent regressions in systemd tests where its stated that the cgroups are set up in "unified" (v2) mode in 248.3-1ubuntu5 systemd package. 248.3-1ubuntu7 became the release package on 2021-09-27 which was around the start of the cycle that we started seeing this issues (did we see this before then?), which means that it could very likely have been this upgrade that caused it. Even if earlier packages were in updates I am pretty sure that the machine needs to be restarted for the systemd changes to take affect, which I am pretty sure our test machines don't restart after updating during deployment. So it would make sense that we are seeing the errors correlating with the new systemd package hitting release.

I also re-ran tests on previous kernels which didnt have these issues and they are now reporting all the errors, which also points to this being a problem with the systemd package upgrade.

In the meantime, it seems that these tests are broken and we should just hint the whole ubuntu_ltp_controllers test suite.

[Ian May (ian-may)]

Found on impish/linux-aws: 5.13.0-1007.8

[Ian May (ian-may)]

Found on impish/linux-azure: 5.13.0-1008.9

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Po-Hsu Lin (cypressyew)]

Looks like this is not failing on Impish Oracle.

For other kernels, failing cases are:
  * memcg_regression
  * memcg_failcnt
  * memcg_force_empty
  * memcg_limit_in_bytes
  * memcg_stat_rss
  * memcg_subgroup_charge
  * memcg_max_usage_in_bytes
  * memcg_move_charge_at_immigrate
  * memcg_memsw_limit_in_bytes
  * memcg_stat
  * memcg_use_hierarchy
  * memcg_usage_in_bytes
  * memcg_stress
  * memcg_control
  * cgroup_fj_function_cpuset
  * cgroup_fj_function_cpu
  * cgroup_fj_function_memory
  * cgroup_fj_function_blkio
  * cgroup_fj_stress_cpuset_*
  * cgroup_fj_stress_cpu_*
  * cgroup_fj_stress_memory_*
  * cgroup_fj_stress_blkio_*
  * cpuset_regression_test
  * pids_*

Here is the complete test report from bug 1949079
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1949079/+attachment/5536760/+files/sru-gcp-gcp-e2-standard-2-ubuntu_ltp_controllers-log.txt

[Po-Hsu Lin (cypressyew)]

This is failing on impish oracle now.

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Po-Hsu Lin (cypressyew)]

Fixed with LTP fork update (bug 1982995)

With LTP fork updated, the ubuntu_ltp_controllers on Jammy is now failing with:
      * LP: #1986636 cpuset_inherit|cpuset_base_ops|cpuset_exclusive|cpuset_hierarchy|
                     cpuset_syscall|cpuset_sched_domains|cpuset_load_balance|
                     cpuset_hotplug|cpuset_memory|cpuset_memory_pressure
      * LP: #1986638 cpuset_regression_test