memcg_subgroup_charge from controllers in ubuntu_ltp failed on D/E/F/G/I

[Po-Hsu Lin]

Issue found on Azure Disco kernel 5.0.0-1023.24

Failed on instance:
  * Standard_GS2
  * Standard_D16s_v3

Passed on instance:
  * Standard_F32s_v2
  * Standard_L8s_v2
  * Standard_L4s
  * Standard_L8s_v2

(Although it has passed on these instances, but they all have the same test output, just the pids are different)

Test failed with:
  /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error

 tag=memcg_subgroup_charge stime=1570239843 dur=3 exit=exited stat=1 core=no cu=7 cs=11
 startup='Sat Oct 5 01:44:03 2019'
 memcg_subgroup_charge 1 TINFO: Starting test 1
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 1 TINFO: Warming up pid: 119041
 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 119041
 memcg_subgroup_charge 1 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 2 TPASS: rss is 0 as expected
 memcg_subgroup_charge 3 TINFO: Starting test 2
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 3 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 3 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 3 TINFO: Warming up pid: 119071
 memcg_subgroup_charge 3 TINFO: Process is still here after warm up: 119071
 memcg_subgroup_charge 3 TFAIL: rss is 0, 135168 expected
 memcg_subgroup_charge 4 TPASS: rss is 0 as expected
 memcg_subgroup_charge 5 TINFO: Starting test 3
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 5 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 5 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 5 TINFO: Warming up pid: 119094
 memcg_subgroup_charge 5 TINFO: Process is still here after warm up: 119094
 memcg_subgroup_charge 5 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 6 TPASS: rss is 0 as expected

[Po-Hsu Lin (cypressyew)]

Issue found on D-kvm:

 startup='Fri Nov 15 03:59:55 2019'
 memcg_subgroup_charge 1 TINFO: Starting test 1
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 1 TINFO: Warming up pid: 712
 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 712
 memcg_subgroup_charge 1 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 2 TPASS: rss is 0 as expected
 memcg_subgroup_charge 3 TINFO: Starting test 2
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 3 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 3 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 3 TINFO: Warming up pid: 738
 memcg_subgroup_charge 3 TINFO: Process is still here after warm up: 738
 memcg_subgroup_charge 3 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 4 TPASS: rss is 0 as expected
 memcg_subgroup_charge 5 TINFO: Starting test 3
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 5 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 5 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 5 TINFO: Warming up pid: 761
 memcg_subgroup_charge 5 TINFO: Process is still here after warm up: 761
 memcg_subgroup_charge 5 TBROK: timeouted on memory.usage_in_bytes
 tag=memcg_subgroup_charge stime=1573790395 dur=13 exit=exited stat=2 core=no cu=33 cs=22

[Po-Hsu Lin (cypressyew)]

Can be found on D-Oracle as well.

[Po-Hsu Lin (cypressyew)]

Spotted on E-5.3 AWS ARM64 instance a1.large, passed on the other 2 ARM64 instances.

[Po-Hsu Lin (cypressyew)]

Didn't see this on E-AWS 5.3.0-1024.26 ARM64

[Po-Hsu Lin (cypressyew)]

Found on F 5.4.0-49.53 node amaura

 startup='Tue Sep 22 01:41:38 2020'
 memcg_subgroup_charge 1 TINFO: Starting test 1
 /opt/ltp/testcases/bin/memcg_subgroup_charge.sh: 522: echo: echo: I/O error
 memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 1 TINFO: Warming up pid: 961276
 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 961276
 memcg_subgroup_charge 1 TBROK: timeouted on memory.usage_in_bytes
 tag=memcg_subgroup_charge stime=1600738898 dur=12 exit=exited stat=2 core=no cu=7 cs=5

[Po-Hsu Lin (cypressyew)]

BTW for comment above (comment #5)
it's the Focal 5.4 lowlatency kernel, the same node passed with the focal generic kernel

[Po-Hsu Lin (cypressyew)]

Spotted on B-5.4 ARM64
 memcg_subgroup_charge 1 TINFO: Starting test 1
 sh: echo: I/O error
 memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 1 TINFO: Warming up pid: 51116
 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 51116
 memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected

[Po-Hsu Lin (cypressyew)]

This cycle on B-5.4 ARM64 node appleton:
 startup='Fri Dec 18 06:32:54 2020'
 memcg_subgroup_charge 1 TINFO: Starting test 1
 sh: echo: I/O error
 memcg_subgroup_charge 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 1 TINFO: Warming up pid: 48615
 memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 48615
 memcg_subgroup_charge 1 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 2 TPASS: rss is 0 as expected
 memcg_subgroup_charge 3 TINFO: Starting test 2
 sh: echo: I/O error
 memcg_subgroup_charge 3 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 3 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 3 TINFO: Warming up pid: 48640
 memcg_subgroup_charge 3 TINFO: Process is still here after warm up: 48640
 memcg_subgroup_charge 3 TPASS: rss is 135168 as expected
 memcg_subgroup_charge 4 TPASS: rss is 0 as expected
 memcg_subgroup_charge 5 TINFO: Starting test 3
 sh: echo: I/O error
 memcg_subgroup_charge 5 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed
 memcg_subgroup_charge 5 TINFO: Running memcg_process --mmap-anon -s 135168
 memcg_subgroup_charge 5 TINFO: Warming up pid: 48667
 memcg_subgroup_charge 5 TINFO: Process is still here after warm up: 48667
 memcg_subgroup_charge 5 TBROK: timeouted on memory.usage_in_bytes
 tag=memcg_subgroup_charge stime=1608273174 dur=14 exit=exited stat=2 core=no cu=24 cs=18

[Guilherme G. Piccoli (gpiccoli)]

Found this on groovy:linux-kvm, cycle sru-20210315.

[Po-Hsu Lin (cypressyew)]

Found on F-oem-5.10 5.10.0-1025.26-oem rumford

[Guilherme G. Piccoli (gpiccoli)]

Observed in H/KVM, cycle sru-20210510.

[Po-Hsu Lin (cypressyew)]

Found on 5.11.0-1005.5 - intel
on node bavor (passed on spitfire)

Test timeout:
  memcg_subgroup_charge 1 TBROK: timed out on memory.usage_in_bytes

Need a manual test to see if we need a longer timeout for it.

[Krzysztof Kozlowski (krzk)]

memcg_subgroup_charge 3 TFAIL: rss is 0, 135168 expected

Found on focal/oracle 5.8.0-1032.33~20.04.1

[Krzysztof Kozlowski (krzk)]

Sent fix to upstream (LTP) for:
memcg_subgroup_charge 3 TFAIL: rss is 0, 135168 expected

There might be here second/different issue reported (memcg_subgroup_charge 1 TBROK: timed out on memory.usage_in_bytes).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Krzysztof Kozlowski (krzk)]

Sent patch to upstream (LTP): https://lists.linux.it/pipermail/ltp/2021-June/023254.html

[Kleber Sacilotto de Souza (kleber-souza)]

Also found with focal/linux-hwe-5.11 5.11.0-20.21~20.04.1.

[Krzysztof Kozlowski (krzk)]

v4 of patchset https://lists.linux.it/pipermail/ltp/2021-July/023764.html applied locally, so marking as fix-released.