vrf_route_leaking.sh in net from ubuntu_kernel_selftests linux ADT test failure with linux/5.11.0-18.19 (Ping received ICMP Packet too big)

[Kelsey Steele]

This is a scripted bug report about ADT failures while running linux tests for linux/5.11.0-18.19 on hirsute. Whether this is caused by the dep8 tests of the tested source or the kernel has yet to be determined.

Not a regression. Found to occur previously on hirsute/linux 5.11.0-14.15

Testing failed on:
    amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/amd64/l/linux/20210515_005957_75e5a@/log.gz
    arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/arm64/l/linux/20210513_203508_96fd3@/log.gz
    ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/ppc64el/l/linux/20210513_163708_c0203@/log.gz
    s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-hirsute/hirsute/s390x/l/linux/20210513_144454_54b04@/log.gz

00:09:30 DEBUG| [stdout] # selftests: net: vrf_route_leaking.sh
00:09:30 DEBUG| [stdout] #
00:09:30 DEBUG| [stdout] # ###########################################################################
00:09:30 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP ttl error route lookup ping
00:09:30 DEBUG| [stdout] # ###########################################################################
00:09:30 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:32 DEBUG| [stdout] # TEST: Ping received ICMP ttl exceeded [ OK ]
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP error route lookup traceroute
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # SKIP: Could not run IPV4 test without traceroute
00:09:32 DEBUG| [stdout] #
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] # IPv4 (sym route): VRF ICMP fragmentation error route lookup ping
00:09:32 DEBUG| [stdout] # ###########################################################################
00:09:32 DEBUG| [stdout] #
00:09:34 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:34 DEBUG| [stdout] # TEST: Ping received ICMP Frag needed [ OK ]
00:09:34 DEBUG| [stdout] #
00:09:34 DEBUG| [stdout] # ###########################################################################
00:09:34 DEBUG| [stdout] # IPv4 (asym route): VRF ICMP ttl error route lookup ping
00:09:34 DEBUG| [stdout] # ###########################################################################
00:09:34 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # TEST: Basic IPv4 connectivity [ OK ]
00:09:36 DEBUG| [stdout] # TEST: Ping received ICMP ttl exceeded [ OK ]
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] # IPv4 (asym route): VRF ICMP error route lookup traceroute
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # SKIP: Could not run IPV4 test without traceroute
00:09:36 DEBUG| [stdout] #
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP ttl error route lookup ping
00:09:36 DEBUG| [stdout] # ###########################################################################
00:09:36 DEBUG| [stdout] #
00:09:40 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [FAIL]
00:09:40 DEBUG| [stdout] # TEST: Ping received ICMP Hop limit [ OK ]
00:09:40 DEBUG| [stdout] #
00:09:40 DEBUG| [stdout] # ###########################################################################
00:09:40 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP error route lookup traceroute
00:09:40 DEBUG| [stdout] # ###########################################################################
00:09:40 DEBUG| [stdout] #
00:09:42 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:42 DEBUG| [stdout] # TEST: Traceroute6 reports a hop on r1 [ OK ]
00:09:42 DEBUG| [stdout] #
00:09:42 DEBUG| [stdout] # ###########################################################################
00:09:42 DEBUG| [stdout] # IPv6 (sym route): VRF ICMP fragmentation error route lookup ping
00:09:42 DEBUG| [stdout] # ###########################################################################
00:09:42 DEBUG| [stdout] #
00:09:45 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [FAIL]
00:09:47 DEBUG| [stdout] # TEST: Ping received ICMP Packet too big [FAIL]
00:09:47 DEBUG| [stdout] #
00:09:47 DEBUG| [stdout] # ###########################################################################
00:09:47 DEBUG| [stdout] # IPv6 (asym route): VRF ICMP ttl error route lookup ping
00:09:47 DEBUG| [stdout] # ###########################################################################
00:09:47 DEBUG| [stdout] #
00:09:49 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:49 DEBUG| [stdout] # TEST: Ping received ICMP Hop limit [ OK ]
00:09:49 DEBUG| [stdout] #
00:09:49 DEBUG| [stdout] # ###########################################################################
00:09:49 DEBUG| [stdout] # IPv6 (asym route): VRF ICMP error route lookup traceroute
00:09:49 DEBUG| [stdout] # ###########################################################################
00:09:49 DEBUG| [stdout] #
00:09:52 DEBUG| [stdout] # TEST: Basic IPv6 connectivity [ OK ]
00:09:52 DEBUG| [stdout] # TEST: Traceroute6 reports a hop on r1 [ OK ]
00:09:52 DEBUG| [stdout] #
00:09:52 DEBUG| [stdout] # Tests passed: 13
00:09:52 DEBUG| [stdout] # Tests failed: 3
00:09:52 DEBUG| [stdout] not ok 45 selftests: net: vrf_route_leaking.sh # exit=1

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1928890

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

This issue can be found on OEM-5.10 as well.

 # IPv6 (sym route): VRF ICMP fragmentation error route lookup ping
 # ###########################################################################
 #
 # TEST: Basic IPv6 connectivity [ OK ]
 # TEST: Ping received ICMP Packet too big [FAIL]

5.10.0-1031.32-oem

[Po-Hsu Lin (cypressyew)]

5.10.0-1035.36 OEM node vought

[Kleber Sacilotto de Souza (kleber-souza)]

Found with hirsute/linux 5.11.0-24.25.

[Po-Hsu Lin (cypressyew)]

Found on 5.11.0-1013.13-aws

[Stefan Bader (smb)]

Found with focal:linux-hwe-5.11 (5.11.0-24.25~20.04.1) on SRU-20210621.

[Krzysztof Kozlowski (krzk)]

Found also on d2021.10.26/jammy/linux-realtime/5.15.0-1003.3

[Krzysztof Kozlowski (krzk)]

Found also on 2021.11.29/impish/linux-oracle/5.13.0-1011.13

[Andrea Righi (arighi)]

This issue is still present also on jammy 5.15. Some details that I found:

  - the purpose of this test is to verify that an ICMP6 packet is not forwarded by a virtual router if it needs to do fragmentation
  - the test creates 3 network namespaces representing host1, host2 and router
  - host1 and host2 have mtu 1500, router has mtu 1400
  - the test consists of a ping6 from host1 to host2 (via router) using a payload of 1450, so host1 and host2 don't need to fragment the packet, but router does
  - therefore the ping is expected to fail (and this is happennig), but we should also receive a "Packet too big" message from the router and we don't

So at the end the behavior is correct, but we're missing the "Packet too big" notification from ping. I'm not sure if it's a user-space issue (in iputils-ping) or a kernel issue.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Zachary Tahenakos (ztahenakos)]

Running this test on jammy 5.15, it passes if the vrf module is missing, the ping will come back with the expected response of Packet too big. With the vrf module loaded, we won't get the message back. Looking at the strace of the command working vs failing, this is a difference spotted:

vrf loaded:
recvmsg(3, {msg_namelen=128}, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(3, {msg_namelen=128}, 0) = -1 EINTR (Interrupted system call)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---

vrf missing:
recvmsg(3, {msg_namelen=128}, 0) = -1 EMSGSIZE (Message too long)

[Zachary Tahenakos (ztahenakos)]

Doing some more research and testing, looking at the upstream commit that added the test (1a01727676a8), this is in the bottom of the message:

    [ The IPv6 MTU test still fails with the symmetric routing setup. It
      appears to be caused by source address selection picking ::1. Fixing
      this is beyond the scope of this series. ]

Running the test in the failing state, we are indeed getting a Packet too big message but we never seem to get it.

Now, within the create_vrf function, if we change the address added from ::1 to ::3
(ip -netns "${ns}" -6 addr add ::3 dev "${vrf}" nodad), it will start to work. I wonder if its getting confused since in the failing case, the src is ::1 and the dest is 2001:db8:16:1:1 so maybe the packet is going around in a loop or something.

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is still failing on recent kernels.

Jammy 5.15:
https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/l/linux/20230327_235241_c3754@/log.gz

Kinetic 5.19:
https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/l/linux/20230328_203949_367cf@/log.gz

[Po-Hsu Lin (cypressyew)]

Found on J-oem-6.5.0-1002