Intel AX201 / Thinkpad X1 Carbon Gen 9 returns ping during WiFi roam

[agoodm]

My Thinkpad X1 Carbon Gen 9 has started to return pings in the middle of WiFi roam events which is causing the IP address to change every time it roams on the WiFi network. This is a huge problem for me because I install WiFi networks for a living but in general it causes every TCP connection over IPv4 to die every time the laptop roams to a new wifi cell.

00:14.3 Network controller: Intel Corporation Wi-Fi 6 AX201 (rev 20)

I tried kernel 5.15.0-40-generic and mainline kernel 5.17 and 5.18.

Jul 11 20:11:15 littlebrat kernel: [620752.349053] wlp0s20f3: authenticate with 86:2a:a8:8b:05:cb
Jul 11 20:11:15 littlebrat kernel: [620752.349074] wlp0s20f3: Invalid HE elem, Disable HE
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0110] device (wlp0s20f3): supplicant interface state: completed -> authenticating
Jul 11 20:11:15 littlebrat kernel: [620752.354373] wlp0s20f3: send auth to 86:2a:a8:8b:05:cb (try 1/3)
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0111] device (p2p-dev-wlp0s20f3): supplicant management interface state: completed -> authenticating
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0111] device (wlp0s20f3): ip:dhcp4: restarting
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0112] dhcp4 (wlp0s20f3): canceled DHCP transaction
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0112] dhcp4 (wlp0s20f3): activation: beginning transaction (timeout in 45 seconds)
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0112] dhcp4 (wlp0s20f3): state changed no lease
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0113] dhcp4 (wlp0s20f3): activation: beginning transaction (timeout in 45 seconds)
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: Trying to associate with 86:2a:a8:8b:05:cb (SSID='YesComputerSolutions-Private' freq=5540 MHz)
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0463] device (wlp0s20f3): supplicant interface state: authenticating -> associating
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0464] device (p2p-dev-wlp0s20f3): supplicant management interface state: authenticating -> associating
Jul 11 20:11:15 littlebrat kernel: [620752.390520] wlp0s20f3: authenticated
Jul 11 20:11:15 littlebrat kernel: [620752.393074] wlp0s20f3: associate with 86:2a:a8:8b:05:cb (try 1/3)
Jul 11 20:11:15 littlebrat kernel: [620752.397915] wlp0s20f3: RX ReassocResp from 86:2a:a8:8b:05:cb (capab=0x1511 status=0 aid=4)
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: Associated with 86:2a:a8:8b:05:cb
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
Jul 11 20:11:15 littlebrat kernel: [620752.406395] wlp0s20f3: associated
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0779] device (wlp0s20f3): supplicant interface state: associating -> associated
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0779] device (p2p-dev-wlp0s20f3): supplicant management interface state: associating -> associated
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0799] device (wlp0s20f3): supplicant interface state: associated -> 4way_handshake
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0800] device (p2p-dev-wlp0s20f3): supplicant management interface state: associated -> 4way_handshake
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: WPA: Key negotiation completed with 86:2a:a8:8b:05:cb [PTK=CCMP GTK=CCMP]
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: CTRL-EVENT-CONNECTED - Connection to 86:2a:a8:8b:05:cb completed [id=0 id_str=]
Jul 11 20:11:15 littlebrat discord_discord.desktop[4119]: [2022-07-11 20:11:15.091] [4119] (discord.cpp:551): JS console: ["%c[GatewaySocket]","Performing an expedited heartbeat reason: network detected online."]
Jul 11 20:11:15 littlebrat wpa_supplicant[1550]: wlp0s20f3: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-66 noise=9999 txrate=26000
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0970] device (wlp0s20f3): supplicant interface state: 4way_handshake -> completed
Jul 11 20:11:15 littlebrat NetworkManager[1519]: <info> [1657566675.0975] device (p2p-dev-wlp0s20f3): supplicant management interface state: 4way_handshake -> completed

Jul 11 20:11:17 gateway-ycs dhcpd: DHCPDISCOVER from 5c:e4:2a:c3:f9:42 (littlebrat) via br_v251
Jul 11 20:11:17 gateway-ycs dhcpd: Abandoning IP address 172.16.251.221: pinged before offer
Jul 11 20:11:19 gateway-ycs dhcpd: DHCPDISCOVER from 5c:e4:2a:c3:f9:42 via br_v251
Jul 11 20:11:20 gateway-ycs dhcpd: DHCPOFFER on 172.16.251.220 to 5c:e4:2a:c3:f9:42 (littlebrat) via br_v251
Jul 11 20:11:20 gateway-ycs dhcpd: DHCPREQUEST for 172.16.251.220 (172.16.251.254) from 5c:e4:2a:c3:f9:42 (littlebrat) via br_v251
Jul 11 20:11:20 gateway-ycs dhcpd: DHCPACK on 172.16.251.220 to 5c:e4:2a:c3:f9:42 (littlebrat) via br_v251

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-tools-5.15.0-40 (not installed)
ProcVersionSignature: Ubuntu 5.15.0-40.43-generic 5.15.35
Uname: Linux 5.15.0-40-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: alan 2413 F.... pulseaudio
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Mon Jul 11 21:49:21 2022
InstallationDate: Installed on 2022-06-05 (36 days ago)
InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Release amd64 (20220419)
MachineType: LENOVO 20XWCT01WW
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-40-generic root=/dev/mapper/vgubuntu-root ro quiet splash
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-40-generic N/A
 linux-backports-modules-5.15.0-40-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.2
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/08/2022
dmi.bios.release: 1.52
dmi.bios.vendor: LENOVO
dmi.bios.version: N32ET76W (1.52 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20XWCT01WW
dmi.board.vendor: LENOVO
dmi.board.version: NO DPK
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.ec.firmware.release: 1.32
dmi.modalias: dmi:bvnLENOVO:bvrN32ET76W(1.52):bd04/08/2022:br1.52:efr1.32:svnLENOVO:pn20XWCT01WW:pvrThinkPadX1CarbonGen9:rvnLENOVO:rn20XWCT01WW:rvrNODPK:cvnLENOVO:ct10:cvrNone:skuLENOVO_MT_20XW_BU_Think_FM_ThinkPadX1CarbonGen9:
dmi.product.family: ThinkPad X1 Carbon Gen 9
dmi.product.name: 20XWCT01WW
dmi.product.sku: LENOVO_MT_20XW_BU_Think_FM_ThinkPad X1 Carbon Gen 9
dmi.product.version: ThinkPad X1 Carbon Gen 9
dmi.sys.vendor: LENOVO

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[agoodm (alan-goodmanemail)]

I edited /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf changing wifi.powersave = 3 to wifi.powersave = 2.

This improved the stability of the WiFi and appears to have stopped the continuous 'roaming'. I will keep an eye on the dhcp logs at home and report back if the 'ping during roam' continues.

[agoodm (alan-goodmanemail)]

To update this, powersave to 2 does not prevent the icmp during roam problems.

I set net.ipv4.icmp_echo_ignore_all=1 in /etc/sysctl.conf as a workaround.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kevin Pulo (devkev)]

I have the same symptoms on Ubuntu 22.04.1 on a 12th-gen Framework laptop with Intel Wi-Fi 6 AX210/AX211/AX411 160MHz (rev 1a) device. Every time wpa-supplicant chooses to associate with a different BSSID (of the same SSID), the interface ends up being given a new IP address by the DHCP server (usually the sequentially next address).

I don't have access to the DHCP server or its logs, but I tried the above suggested workaround of setting net.ipv4.icmp_echo_ignore_all=1 and this prevents the problem from occurring. So I expect the root cause is the same. (And as an aside, there is nothing in the client-side logs to indicate that ICMP pings could be in any way related; without this report it would have much more difficult to diagnose and work around the problem.)

A different laptop running 18.04 on the same network is able to keep its IP address when roaming, so presumably this is due to a change in NetworkManager behaviour.

The problem makes it difficult to work in an enterprise office environment (ie. moving around between desks and meeting rooms), because all TCP connections are lost when the IP address changes, which is especially impactful for ssh connections.

[Kevin Pulo (devkev)]

The only upstream bug that seems potentially related is https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1024 . But that's more about ensuring that WPA auth has completed before starting the DHCP renewal, and it's not clear to me how blocking ICMP could avoid that problem. So it might not actually be related. The fix for that issue went into development version NM 1.39.9, so I may try building and trying an upstream version which includes it, to see if that fixes the problem (but I won't be able to try before next week at the earliest).

[agoodm (alan-goodmanemail)]

The reason blocking pings prevents the IP address change is because a lot of DHCP servers ping the IP before they (re)allocate the address. If the device responds to the ping then the DHCP server abandons the IP and leases a new one.

For this reason the network stack should; in my opinion ignore IPv4 ICMP echo requests during the DHCP part of the roam process. Indeed I believe that this was the behaviour previously.

Blocking IPv4 pings isnt impacting any other functionality for me.