Activity log for bug #1933969

Date Who What changed Old value New value Message
2021-06-29 10:47:12 Po-Hsu Lin bug added bug
2021-06-29 10:55:12 Po-Hsu Lin summary ubuntu_kernel_selftests will unable to finish properly on Groovy s390x net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy s390x
2021-06-29 10:55:28 Po-Hsu Lin tags 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests
2021-06-29 11:00:35 Po-Hsu Lin description Issue found on kernel03 zVM with Groovy 5.8.0-60.67 On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? If you take a closer look on this node, you will see it's complaining modprobe command hung and killed: unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 INFO: task modprobe:40237 blocked for more than 122 seconds. Tainted: P O 5.8.0-60-generic #67-Ubuntu "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. modprobe D 0 40237 1265 0x00000000 Call Trace: [<0000000036dc7860>] __schedule+0x300/0x840 [<0000000036dc7e02>] schedule+0x62/0x100 [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0 [<0000000036b3716c>] register_pernet_device+0x2c/0xa0 [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti] [<00000000361a88b0>] do_one_initcall+0x40/0x200 [<000000003632b34e>] do_init_module+0x6e/0x530 [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0 [<0000000036dcefbc>] system_call+0xe0/0x2b8 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows: root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10 root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti From the code it looks like this is caused by the net/pmtu.sh Issue found on kernel03 zVM with Groovy 5.8.0-60.67 The pmtu.sh test will hang, and gets killed: # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ] # TEST: vti6: PMTU exceptions [ OK ] Timer expired (1800 sec.), nuking pid 13090 If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung: unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 INFO: task modprobe:40237 blocked for more than 122 seconds. Tainted: P O 5.8.0-60-generic #67-Ubuntu "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. modprobe D 0 40237 1265 0x00000000 Call Trace: [<0000000036dc7860>] __schedule+0x300/0x840 [<0000000036dc7e02>] schedule+0x62/0x100 [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0 [<0000000036b3716c>] register_pernet_device+0x2c/0xa0 [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti] [<00000000361a88b0>] do_one_initcall+0x40/0x200 [<000000003632b34e>] do_init_module+0x6e/0x530 [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0 [<0000000036dcefbc>] system_call+0xe0/0x2b8 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows: root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10 root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh This will prevent the whole ubuntu_kernel_selftests to finish On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me.
2021-06-29 11:00:41 Po-Hsu Lin summary net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy s390x test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy s390x
2021-06-29 11:02:33 Po-Hsu Lin bug task added linux (Ubuntu)
2021-06-29 11:02:43 Po-Hsu Lin nominated for series Ubuntu Groovy
2021-06-29 11:02:43 Po-Hsu Lin bug task added linux (Ubuntu Groovy)
2021-06-29 11:04:25 Po-Hsu Lin description Issue found on kernel03 zVM with Groovy 5.8.0-60.67 The pmtu.sh test will hang, and gets killed: # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ] # TEST: vti6: PMTU exceptions [ OK ] Timer expired (1800 sec.), nuking pid 13090 If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung: unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 INFO: task modprobe:40237 blocked for more than 122 seconds. Tainted: P O 5.8.0-60-generic #67-Ubuntu "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. modprobe D 0 40237 1265 0x00000000 Call Trace: [<0000000036dc7860>] __schedule+0x300/0x840 [<0000000036dc7e02>] schedule+0x62/0x100 [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0 [<0000000036b3716c>] register_pernet_device+0x2c/0xa0 [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti] [<00000000361a88b0>] do_one_initcall+0x40/0x200 [<000000003632b34e>] do_init_module+0x6e/0x530 [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0 [<0000000036dcefbc>] system_call+0xe0/0x2b8 unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows: root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10 root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh This will prevent the whole ubuntu_kernel_selftests to finish On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me. Issue found on kernel03 zVM with Groovy 5.8.0-60.67 Test case: * tools/testing/selftests/net/pmtu.sh The pmtu.sh test will hang, and gets killed: # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ] # TEST: vti6: PMTU exceptions [ OK ] Timer expired (1800 sec.), nuking pid 13090 If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung:   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   INFO: task modprobe:40237 blocked for more than 122 seconds.         Tainted: P O 5.8.0-60-generic #67-Ubuntu   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.   modprobe D 0 40237 1265 0x00000000   Call Trace:    [<0000000036dc7860>] __schedule+0x300/0x840    [<0000000036dc7e02>] schedule+0x62/0x100    [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0    [<0000000036b3716c>] register_pernet_device+0x2c/0xa0    [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti]    [<00000000361a88b0>] do_one_initcall+0x40/0x200    [<000000003632b34e>] do_init_module+0x6e/0x530    [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0    [<0000000036dcefbc>] system_call+0xe0/0x2b8   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows:   root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10   root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh This will prevent the whole ubuntu_kernel_selftests to finish On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me.
2021-06-29 11:04:30 Po-Hsu Lin summary test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy s390x test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy
2021-06-29 11:05:28 Po-Hsu Lin description Issue found on kernel03 zVM with Groovy 5.8.0-60.67 Test case: * tools/testing/selftests/net/pmtu.sh The pmtu.sh test will hang, and gets killed: # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ] # TEST: vti6: PMTU exceptions [ OK ] Timer expired (1800 sec.), nuking pid 13090 If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung:   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   INFO: task modprobe:40237 blocked for more than 122 seconds.         Tainted: P O 5.8.0-60-generic #67-Ubuntu   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.   modprobe D 0 40237 1265 0x00000000   Call Trace:    [<0000000036dc7860>] __schedule+0x300/0x840    [<0000000036dc7e02>] schedule+0x62/0x100    [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0    [<0000000036b3716c>] register_pernet_device+0x2c/0xa0    [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti]    [<00000000361a88b0>] do_one_initcall+0x40/0x200    [<000000003632b34e>] do_init_module+0x6e/0x530    [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0    [<0000000036dcefbc>] system_call+0xe0/0x2b8   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows:   root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10   root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh This will prevent the whole ubuntu_kernel_selftests to finish On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me. Issue found with Groovy 5.8.0-60.67 on different amd64 / arm64 / s390x instances Test case: * tools/testing/selftests/net/pmtu.sh The pmtu.sh test will hang, and gets killed: # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ] # TEST: vti6: PMTU exceptions [ OK ] Timer expired (1800 sec.), nuking pid 13090 If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung:   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1   INFO: task modprobe:40237 blocked for more than 122 seconds.         Tainted: P O 5.8.0-60-generic #67-Ubuntu   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.   modprobe D 0 40237 1265 0x00000000   Call Trace:    [<0000000036dc7860>] __schedule+0x300/0x840    [<0000000036dc7e02>] schedule+0x62/0x100    [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0    [<0000000036b3716c>] register_pernet_device+0x2c/0xa0    [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti]    [<00000000361a88b0>] do_one_initcall+0x40/0x200    [<000000003632b34e>] do_init_module+0x6e/0x530    [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0    [<0000000036dcefbc>] system_call+0xe0/0x2b8   unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1 Running process shows:   root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10   root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh This will prevent the whole ubuntu_kernel_selftests to finish On Jenkins, the test will hang and killed by the timeout: 10:08:18 INFO | ========== Summary =========== 10:08:18 INFO | No failed cases reported 10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully 10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 1 10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted 10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18 10:08:18 DEBUG| Persistent state client._record_indent now set to 0 10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html Build timed out (after 240 minutes). Marking the build as aborted. Build was aborted Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me.
2021-06-29 11:44:32 Guilherme G. Piccoli bug added subscriber Guilherme G. Piccoli
2021-06-29 13:14:00 Stefan Bader bug added subscriber Stefan Bader
2021-06-29 14:06:07 Stefan Bader nominated for series Ubuntu Hirsute
2021-06-29 14:06:07 Stefan Bader bug task added linux (Ubuntu Hirsute)
2021-06-29 14:06:26 Stefan Bader linux (Ubuntu Groovy): importance Undecided High
2021-06-29 14:06:26 Stefan Bader linux (Ubuntu Groovy): status New Triaged
2021-06-29 14:06:40 Stefan Bader linux (Ubuntu Hirsute): importance Undecided High
2021-06-29 14:06:40 Stefan Bader linux (Ubuntu Hirsute): status New Triaged
2021-06-29 14:06:47 Stefan Bader linux (Ubuntu): status New Invalid
2021-06-29 14:07:24 Stefan Bader summary test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8
2021-06-30 06:29:08 Stefan Bader linux (Ubuntu Hirsute): status Triaged Fix Committed
2021-06-30 06:29:44 Stefan Bader linux (Ubuntu Hirsute): assignee Stefan Bader (smb)
2021-06-30 06:29:48 Stefan Bader linux (Ubuntu Groovy): assignee Stefan Bader (smb)
2021-06-30 08:26:06 Stefan Bader linux (Ubuntu Groovy): status Triaged Fix Committed
2021-07-06 14:06:40 Stefan Bader tags 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests verification-needed-groovy verification-needed-hirsute
2021-07-07 12:13:42 Stefan Bader tags 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests verification-needed-groovy verification-needed-hirsute 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests verification-done-groovy verification-needed-hirsute
2021-07-08 08:39:48 Stefan Bader tags 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests verification-done-groovy verification-needed-hirsute 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests verification-done-groovy verification-done-hirsute
2021-07-20 16:18:41 Launchpad Janitor linux (Ubuntu Hirsute): status Fix Committed Fix Released
2021-07-20 16:18:41 Launchpad Janitor cve linked 2021-33909
2021-07-20 16:20:45 Launchpad Janitor linux (Ubuntu Groovy): status Fix Committed Fix Released
2022-12-30 10:38:59 Po-Hsu Lin ubuntu-kernel-tests: status New Fix Released