mem-on-off-test.sh from memory-hotplug in ubuntu_kernel_selftests failed on X-gcp-4.15 / F-5.4 zVM

[Po-Hsu Lin]

Issue found on Focal 5.4.0-49.53 zVM kernel04

Test failed with:
 # selftests: memory-hotplug: mem-on-off-test.sh
 # Test scope: 2% hotplug memory
 # online all hot-pluggable memory in offline state:
 # SKIPPED - no hot-pluggable memory in offline state
 # offline 2% hot-pluggable memory in online state
 # trying to offline 1 out of 16 memory block(s):
 # online->offline memory0
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 0: unexpected fail
 # online->offline memory1
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 1: unexpected fail
 # online->offline memory10
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 10: unexpected fail
 # online->offline memory11
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 11: unexpected fail
 # online->offline memory12
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 12: unexpected fail
 # online->offline memory13
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 13: unexpected fail
 # online->offline memory14
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 14: unexpected fail
 # online->offline memory15
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 15: unexpected fail
 # online->offline memory2
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 2: unexpected fail
 # online->offline memory3
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 3: unexpected fail
 # online->offline memory4
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 4: unexpected fail
 # online->offline memory5
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 5: unexpected fail
 # online->offline memory6
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 6: unexpected fail
 # online->offline memory7
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 7: unexpected fail
 # online->offline memory8
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 8: unexpected fail
 # online->offline memory9
 # ./mem-on-off-test.sh: line 78: echo: write error: Device or resource busy
 # offline_memory_expect_success 9: unexpected fail
 # FAILED - unable to offline some memory blocks, device busy?
 # online all hot-pluggable memory in offline state:
 # SKIPPED - no hot-pluggable memory in offline state
 # Test with memory notifier error injection
 not ok 1 selftests: memory-hotplug: mem-on-off-test.sh # exit=1

This issue can be found on 5.4.0-46.50, 5.4.0-45.49 zVM as well.
Passed with 5.4.0-44.48
Passed with 5.4.0-43.47
Failed with 5.4.0-42.46

Looks like it's not very stable.

[Po-Hsu Lin (cypressyew)]

This "FAILED - unable to offline some memory blocks, device busy?" can be found on some instances (n1-standard-64) with X-gcp 4.15.0-1085.96 as well

[Kelsey Steele (kelsey-steele)]

Found on Focal 'aws : 5.4.0-1026.26 : amd64' on multiple instances

[Francis Ginther (fginther)]

Seen on groovy 5.8 on t2.small.

[Po-Hsu Lin (cypressyew)]

Found on Focal GCP 5.4.0-1036.37

[Kelsey Steele (kelsey-steele)]

spotted on azure : 4.15.0-1106.118 : amd64

[Francis Ginther (fginther)]

Seen with linux-gcp 5.4.0-1037.40.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1897764

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Marcelo Cerri (mhcerri)]

Also seen in xenial linux-fips 4.4.0-1060.66 for sru-20210315

[Krzysztof Kozlowski (krzk)]

38. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # Test with memory notifier error injection
39. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # ./mem-on-off-test.sh: line 267: echo: write error: Invalid argument
40. 02/25 16:08:02 DEBUG| utils:0153| [stdout] # ./mem-on-off-test.sh: line 283: echo: write error: Invalid argument
41. 02/25 16:08:03 DEBUG| utils:0153| [stdout] # offline_memory_expect_fail 13: unexpected success
42. 02/25 16:08:03 DEBUG| utils:0153| [stdout] # offline_memory_expect_fail 14: unexpected success

groovy/aws 5.8.0-1031.33

[Krzysztof Kozlowski (krzk)]

Also on: groovy/oracle 5.8.0-1027.28

[Krzysztof Kozlowski (krzk)]

Found on focal/azure 5.8.0-1034.36~20.04.1-azure

[Krzysztof Kozlowski (krzk)]

Sent patch: https://lists.ubuntu.com/archives/kernel-team/2021-June/121361.html

[Kleber Sacilotto de Souza (kleber-souza)]

This is still affecting at least the generic kernels for amd64 and ppc64el arches.

[Luke Nowakowski-Krijger (lukenow)]

Observed on F/gcp 5.4.0-1053.57 on e2-standard-2, f1-micro, n1-highcpu-4

[Bartlomiej Zolnierkiewicz (bzolnier)]

Observed also on 2021.11.29/focal/linux-gcp/5.4.0-1059.63 on f1-micro and g1-small instances

[Bartlomiej Zolnierkiewicz (bzolnier)]

Also seen in bionic linux-ibm-5.4 5.4.0-1021.23~18.04.1 for sru-20220418