test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8

Bug #1933969 reported by Po-Hsu Lin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
Undecided
Unassigned
linux (Ubuntu)
Undecided
Unassigned
Groovy
High
Stefan Bader
Hirsute
High
Stefan Bader

Bug Description

Issue found with Groovy 5.8.0-60.67 on different amd64 / arm64 / s390x instances

Test case:
* tools/testing/selftests/net/pmtu.sh

The pmtu.sh test will hang, and gets killed:
# TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects [ OK ]
# TEST: vti6: PMTU exceptions [ OK ]
Timer expired (1800 sec.), nuking pid 13090

If you take a closer look on this node, you will see it's complaining about unregister_netdevice and modprobe command hung:
  unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1
  unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1
  INFO: task modprobe:40237 blocked for more than 122 seconds.
        Tainted: P O 5.8.0-60-generic #67-Ubuntu
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  modprobe D 0 40237 1265 0x00000000
  Call Trace:
   [<0000000036dc7860>] __schedule+0x300/0x840
   [<0000000036dc7e02>] schedule+0x62/0x100
   [<00000000362c3bce>] rwsem_down_write_slowpath+0x23e/0x6c0
   [<0000000036b3716c>] register_pernet_device+0x2c/0xa0
   [<000003ff80a16034>] vti_init+0x34/0x1000 [ip_vti]
   [<00000000361a88b0>] do_one_initcall+0x40/0x200
   [<000000003632b34e>] do_init_module+0x6e/0x530
   [<000000003632e06a>] __do_sys_finit_module+0xaa/0xf0
   [<0000000036dcefbc>] system_call+0xe0/0x2b8
  unregister_netdevice: waiting for ip6_vti0 to become free. Usage count = 1

Running process shows:
  root 40236 0.0 0.0 3748 1040 ? D 10:39 0:00 ip link add vti4_a type vti local 192.168.1.1 remote 192.168.1.2 key 10
  root 40237 0.0 0.0 6092 1560 ? D 10:39 0:00 /sbin/modprobe -q -- rtnl-link-vti

This command "ip link add vti4_a type vti local" belongs to the test_pmtu_vti4_link_add_mtu() test in pmtu.sh

This will prevent the whole ubuntu_kernel_selftests to finish

On Jenkins, the test will hang and killed by the timeout:
10:08:18 INFO | ========== Summary ===========
10:08:18 INFO | No failed cases reported
10:08:18 INFO | GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18 completed successfully
10:08:18 INFO | END GOOD ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1624439298 localtime=Jun 23 10:08:18
10:08:18 DEBUG| Persistent state client._record_indent now set to 1
10:08:18 DEBUG| Persistent state client.unexpected_reboot deleted
10:08:18 INFO | END GOOD ---- ---- timestamp=1624439298 localtime=Jun 23 10:08:18
10:08:18 DEBUG| Persistent state client._record_indent now set to 0
10:08:18 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html
Build timed out (after 240 minutes). Marking the build as aborted.
Build was aborted
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?

Since we don't have any change to the pmtu.sh in this cycle. This looks like a potential regression to me.

CVE References

Po-Hsu Lin (cypressyew)
summary: - ubuntu_kernel_selftests will unable to finish properly on Groovy s390x
+ net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy
+ s390x
tags: added: 5.8 groovy kqa-blocker s390x sru-20210621 ubuntu-kernel-selftests
Po-Hsu Lin (cypressyew)
description: updated
summary: - net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy
- s390x
+ test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
+ ubuntu_kernel_selftests cannot finish properly on Groovy s390x
description: updated
summary: test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
- ubuntu_kernel_selftests cannot finish properly on Groovy s390x
+ ubuntu_kernel_selftests cannot finish properly on Groovy
description: updated
Revision history for this message
Guilherme G. Piccoli (gpiccoli) wrote : Re: test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in ubuntu_kernel_selftests cannot finish properly on Groovy

Thanks for the report!! We may be missing this patch:

https://git.kernel.org/linus/0d7a7b2014b1 ("ipv6: remove extra dev_hold() for fallback tunnels")

It has a fixes tag for upstream 40cb881b5aaa, which was merged in our newer 5.8 kernels - can you test with the fix @cypressyew and let us know?
Thanks!

Revision history for this message
Stefan Bader (smb) wrote :

Actually this might need all 4 of those:

5b06a2cd65fe ipv6: remove extra dev_hold() for fallback tunnels
b836b8c0c288 ip6_tunnel: sit: proper dev_{hold|put} in ndo_[un]init methods
49cd360d640f sit: proper dev_{hold|put} in ndo_[un]init methods
b0a78d4b8768 ip6_gre: proper dev_{hold|put} in ndo_[un]init methods

Two of them refer to 40cb881b5aaa but all have a similar message.

Revision history for this message
Stefan Bader (smb) wrote :

To say above is in hirsute. Not sure all 4 are for Groovy, too.

Revision history for this message
Stefan Bader (smb) wrote :

From quick scan: this seems to affect all hirsute and groovy (5.11 and 5.8) kernels. For Focal (5.4) and Bionic (4.15) the follow-up stable commits appear to be applied at least for the 2021.06.21 cycle.

Revision history for this message
Stefan Bader (smb) wrote :

For reference these are the upstream SHA1 references:

0d7a7b2014b1 ipv6: remove extra dev_hold() for fallback tunnels
48bb5697269a ip6_tunnel: sit: proper dev_{hold|put} in ndo_[un]init methods
6289a98f0817 sit: proper dev_{hold|put} in ndo_[un]init methods
7f700334be9a ip6_gre: proper dev_{hold|put} in ndo_[un]init methods

Revision history for this message
Stefan Bader (smb) wrote :

@Sam, I will be uploading a fixup Hirsute kernel (with the 4 additional patches picked) to

https://launchpad.net/~smb/+archive/ubuntu/temp01

That showed the same regression in ADT logs as Groovy did. So if it works there it would help Groovy as well. If you could rerun the linux regression tests on that first. Thanks.

Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in linux (Ubuntu Hirsute):
importance: Undecided → High
status: New → Triaged
Changed in linux (Ubuntu):
status: New → Invalid
summary: test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
- ubuntu_kernel_selftests cannot finish properly on Groovy
+ ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

@Stefan, thanks for the test kernel.
I gave both Groovy / Hirsute kernel a try. These two kernels can fix the test hang issue of the current proposed kernel in G/H respectively.
Thanks!

Stefan Bader (smb)
Changed in linux (Ubuntu Hirsute):
status: Triaged → Fix Committed
assignee: nobody → Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
assignee: nobody → Stefan Bader (smb)
Stefan Bader (smb)
Changed in linux (Ubuntu Groovy):
status: Triaged → Fix Committed
Revision history for this message
Stefan Bader (smb) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Revision history for this message
Stefan Bader (smb) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-hirsute' to 'verification-done-hirsute'. If the problem still exists, change the tag 'verification-needed-hirsute' to 'verification-failed-hirsute'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-groovy verification-needed-hirsute
Revision history for this message
Stefan Bader (smb) wrote :

ADT testing formerly had net subtests being killed and resource busy messages for netdev structures for the 5.8 kernel. The run with the replacement kernel no longer shows those (https://autopkgtest.ubuntu.com/results/autopkgtest-groovy/groovy/amd64/l/linux/20210630_224630_29e20@/log.gz).

tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Stefan Bader (smb) wrote :
tags: added: verification-done-hirsute
removed: verification-needed-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (23.1 KiB)

This bug was fixed in the package linux - 5.11.0-25.27

---------------
linux (5.11.0-25.27) hirsute; urgency=medium

  * CVE-2021-33909
    - SAUCE: seq_file: Disallow extremely large seq buffer allocations

linux (5.11.0-24.25) hirsute; urgency=medium

  * test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
    ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8
    (LP: #1933969)
    - ip6_gre: proper dev_{hold|put} in ndo_[un]init methods
    - sit: proper dev_{hold|put} in ndo_[un]init methods
    - ip6_tunnel: sit: proper dev_{hold|put} in ndo_[un]init methods
    - ipv6: remove extra dev_hold() for fallback tunnels

linux (5.11.0-23.24) hirsute; urgency=medium

  * hirsute/linux: 5.11.0-23.24 -proposed tracker (LP: #1932420)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - update dkms package versions

  * Disable hv-kvp-daemon.service on certain instance types (LP: #1932081)
    - [Packaging]: Add kernel command line condition to hv-kvp-daemon service

  * Add support for IO functions of AAEON devices (LP: #1929504)
    - ODM: mfd: Add support for IO functions of AAEON devices
    - ODM: gpio: add driver for AAEON devices
    - ODM: watchdog: add driver for AAEON devices
    - ODM: hwmon: add driver for AAEON devices
    - ODM: leds: add driver for AAEON devices
    - ODM: [Config] update config for AAEON devices

  * Add support for selective build of special drivers (LP: #1912789)
    - [Packaging] Add support for ODM drivers
    - [Packaging] Turn on ODM support for amd64
    - [Packaging] Fix ODM support in actual build
    - [Packaging] Fix ODM DRIVERS Kconfig

  * Mute/Mic mute LEDs and right speaker are not work on HP platforms
    (LP: #1932055)
    - ALSA: hda/realtek: fix mute/micmute LEDs and speaker for HP Elite Dragonfly
      G2
    - ALSA: hda/realtek: fix mute/micmute LEDs and speaker for HP EliteBook x360
      1040 G8
    - ALSA: hda/realtek: fix mute/micmute LEDs for HP EliteBook 840 Aero G8
    - ALSA: hda/realtek: fix mute/micmute LEDs for HP ZBook Power G8

  * SD card initialization on insertion fails (LP: #1929444)
    - misc: rtsx: separate aspm mode into MODE_REG and MODE_CFG

  * Fix non-working GPU on Some HP desktops (LP: #1931147)
    - PCI: Coalesce host bridge contiguous apertures

  * CirrusLogic: The default input volume is "0%" on Dell Warlock (LP: #1929803)
    - ALSA: hda/cirrus: Set Initial DMIC volume to -26 dB

  * Mic-mute/mute LEDs not work on some HP platforms (LP: #1930707)
    - ALSA: hda/realtek: fix mute/micmute LEDs for HP 855 G8
    - ALSA: hda/realtek: fix mute/micmute LEDs and speaker for HP Zbook G8
    - ALSA: hda/realtek: fix mute/micmute LEDs and speaker for HP Zbook Fury 15 G8
    - ALSA: hda/realtek: fix mute/micmute LEDs and speaker for HP Zbook Fury 17 G8

  * [UBUNTU 21.04] tools/kvm_stat: Add restart delay (LP: #1921870)
    - [Packaging] install kvm_stat systemd service

  * Fix ICL PCH no picture after S3 (LP: #1930582)
    - drm/i915/icp+: Use icp_hpd_irq_setup() instead of spt_hpd_irq_setup()

  * Hirsute update: v5.11.22 upstream stable release (LP: #1931292)
    - KEYS: trusted: Fix memory...

Changed in linux (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (60.6 KiB)

This bug was fixed in the package linux - 5.8.0-63.71

---------------
linux (5.8.0-63.71) groovy; urgency=medium

  * Possible io_uring regression with QEMU on Ubuntu's kernel (LP: #1935017)
    - SAUCE: Revert "block: don't ignore REQ_NOWAIT for direct IO"

linux (5.8.0-62.70) groovy; urgency=medium

  * CVE-2021-33909
    - SAUCE: seq_file: Disallow extremely large seq buffer allocations

linux (5.8.0-61.68) groovy; urgency=medium

  * test_pmtu_vti4_link_add_mtu() test from net/pmtu.sh in
    ubuntu_kernel_selftests cannot finish properly on 5.11 and 5.8
    (LP: #1933969)
    - ip6_gre: proper dev_{hold|put} in ndo_[un]init methods
    - sit: proper dev_{hold|put} in ndo_[un]init methods
    - ip6_tunnel: sit: proper dev_{hold|put} in ndo_[un]init methods
    - ipv6: remove extra dev_hold() for fallback tunnels

linux (5.8.0-60.67) groovy; urgency=medium

  * groovy/linux: 5.8.0-60.67 -proposed tracker (LP: #1932438)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - update dkms package versions

  * Add support for IO functions of AAEON devices (LP: #1929504)
    - ODM: mfd: Add support for IO functions of AAEON devices
    - ODM: gpio: add driver for AAEON devices
    - ODM: watchdog: add driver for AAEON devices
    - ODM: hwmon: add driver for AAEON devices
    - ODM: leds: add driver for AAEON devices
    - ODM: [Config] update config for AAEON devices

  * Add support for selective build of special drivers (LP: #1912789)
    - [Packaging] Add support for ODM drivers
    - [Packaging] Turn on ODM support for amd64
    - [Packaging] Fix ODM support in actual build

  * Upstream v5.9 introduced 'module' patches that removed exported symbols
    (LP: #1932065)
    - SAUCE: Revert "modules: inherit TAINT_PROPRIETARY_MODULE"
    - SAUCE: Revert "modules: return licensing information from find_symbol"
    - SAUCE: Revert "modules: rename the licence field in struct symsearch to
      license"
    - SAUCE: Revert "modules: unexport __module_address"
    - SAUCE: Revert "modules: unexport __module_text_address"
    - SAUCE: Revert "modules: mark each_symbol_section static"
    - SAUCE: Revert "modules: mark find_symbol static"
    - SAUCE: Revert "modules: mark ref_module static"

  * Disable hv-kvp-daemon.service on certain instance types (LP: #1932081)
    - [Packaging]: Add kernel command line condition to hv-kvp-daemon service

  * Groovy update: upstream stable patchset 2021-06-10 (LP: #1931637)
    - KEYS: trusted: Fix memory leak on object td
    - tpm: fix error return code in tpm2_get_cc_attrs_tbl()
    - tpm, tpm_tis: Extend locality handling to TPM2 in tpm_tis_gen_interrupt()
    - tpm, tpm_tis: Reserve locality in tpm_tis_resume()
    - KVM: x86/mmu: Remove the defunct update_pte() paging hook
    - PM: runtime: Fix unpaired parent child_count for force_resume
    - kvm: Cap halt polling at kvm->max_halt_poll_ns
    - ath11k: fix thermal temperature read
    - fs: dlm: fix debugfs dump
    - tipc: convert dest node's address to network order
    - ASoC: Intel: bytcr_rt5640: Enable jack-detect support on Asus T100TAF
    - net: stmmac: Set FIFO sizes for ipq806x
   ...

Changed in linux (Ubuntu Groovy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers