Hi Nick,
Thank you for the amazing and timely work!

I checked the kernel versions and you are correct! The versions are different. I don't know how this happened. Is there a way to find out how the kernel version changed?

on the normal machine:
---------
dundts@mongodb-rs3:~$ uname -r
5.15.0-124-generic
dundts@mongodb-rs3:~$ sudo cat /sys/class/net/enp96s0f0/phys_port_name
cat: /sys/class/net/enp96s0f0/phys_port_name: Operation not supported

on the abnormal machine:
---------
dundts@mongodb-rs2:~$ uname -r
6.8.0-47-generic
dundts@mongodb-rs2:~$ sudo cat /sys/class/net/enp96s0f0np0/phys_port_name
p0


________________________________
From: <email address hidden> <email address hidden> on behalf of Nick Rosbrook <email address hidden>
Sent: Wednesday, October 30, 2024 10:15
To: Stoyan Atanasov <email address hidden>
Subject: [Bug 2085835] Re: "Predictable Network Interface Naming" suddenly changed the name from enp96s0f0 to enp96s0f0np0 causing outage

Thanks for the logs!

So, I think in both cases, udev *is* using the 'path' rename policy.
However, it seems that on *one* system, the kernel has a value for
/sys/class/net/<iface>/phys_port_name, which results in the addition to
the interface name.

Can you please run:

$ cat /sys/class/net/<iface>/phys_port_name

on each machine (with the correct interface name), and report back?
Also, can you report the kernel versions (uname -r) from each machine?

My new suspicion is that you have a different/newer kernel on one
machine, and that the network driver gained support for
phys_port_name_show().

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/2085835

Title:
  "Predictable Network Interface Naming" suddenly changed the name from
  enp96s0f0 to enp96s0f0np0 causing outage

Status in systemd package in Ubuntu:
  New

Bug description:
  On three identical servers - sleds in a blade chassis.
  the NIC name of one of the servers suddenly changed, causing an outage. The netplan config could not assign IP to the interface due to the name change.

  enp96s0f0 to enp96s0f0np0

  The two other servers still have the enp96s0f0 name. No BIOS or other
  configuration change is detectable between the three systems.
  Firmware, software - all the same


  I tried to debug this issue myself, but it turned out to be a very niche and complicated topic.
  I went to submit an issue on https://github.com/systemd/systemd/issues, but they only accept issues for newer versions.

  System info:

  dundts@mongodb-rs2:~$ lsb_release -rd
  Description:    Ubuntu 22.04.3 LTS
  Release:        22.04

  dundts@mongodb-rs2:~$ apt-cache policy systemd
  system:
    Installiert:           249.11-0ubuntu3.11
    Installationskandidat: 249.11-0ubuntu3.12
    Versionstabelle:
       249.11-0ubuntu3.12 500
          500 http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
   *** 249.11-0ubuntu3.11 100
          100 /var/lib/dpkg/status
       249.11-0ubuntu3.7 500
          500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
       249.11-0ubuntu3 500
          500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

  dundts@mongodb-rs2:~$ sudo dmesg | grep i40e
  [sudo] password for dundts:
  [    3.657531] i40e: Intel(R) Ethernet Connection XL710 Network Driver
  [    3.658208] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
  [    3.715866] i40e 0000:60:00.0: fw 3.1.54559 api 1.5 nvm 3.2d 0x80000b4b 1.1767.0 [8086:37d3] [152d:8a40]
  [    3.720222] i40e 0000:60:00.0: MAC address: d8:c4:97:4c:66:ae
  [    3.720558] i40e 0000:60:00.0: FW LLDP is enabled
  [    3.732613] i40e 0000:60:00.0: Added LAN device PF0 bus=0x60 dev=0x00 func=0x00
  [    3.733075] i40e 0000:60:00.0: Features: PF-id[0] VFs: 32 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [    3.788671] i40e 0000:60:00.1: fw 3.1.54559 api 1.5 nvm 3.2d 0x80000b4b 1.1767.0 [8086:37d3] [152d:8a40]
  [    3.804478] i40e 0000:60:00.1: MAC address: d8:c4:97:4c:66:af
  [    3.804964] i40e 0000:60:00.1: FW LLDP is enabled
  [    3.839982] i40e 0000:60:00.1 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
  [    3.851939] i40e 0000:60:00.1: Added LAN device PF1 bus=0x60 dev=0x00 func=0x01
  [    3.877994] i40e 0000:60:00.1: Features: PF-id[1] VFs: 32 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [    4.065330] i40e 0000:60:00.0 enp96s0f0np0: renamed from eth0
  [    4.096320] i40e 0000:60:00.1 enp96s0f1np1: renamed from eth1

  dundts@mongodb-rs2:~$ sudo udevadm info /sys/class/net/enp96s0f0np0
  P: /devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/0000:5f:03.0/0000:60:00.0/net/enp96s0f0np0
  L: 0
  E: DEVPATH=/devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/0000:5f:03.0/0000:60:00.0/net/enp96s0f0np0
  E: INTERFACE=enp96s0f0np0
  E: IFINDEX=2
  E: SUBSYSTEM=net
  E: USEC_INITIALIZED=4079509
  E: ID_MM_CANDIDATE=1
  E: ID_NET_NAMING_SCHEME=v249
  E: ID_NET_NAME_MAC=enxd8c4974c66ae
  E: ID_OUI_FROM_DATABASE=Quanta Computer Inc.
  E: ID_NET_NAME_PATH=enp96s0f0np0
  E: ID_BUS=pci
  E: ID_VENDOR_ID=0x8086
  E: ID_MODEL_ID=0x37d3
  E: ID_PCI_CLASS_FROM_DATABASE=Network controller
  E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
  E: ID_VENDOR_FROM_DATABASE=Intel Corporation
  E: ID_MODEL_FROM_DATABASE=Ethernet Connection X722 for 10GbE SFP+
  E: ID_PATH=pci-0000:60:00.0
  E: ID_PATH_TAG=pci-0000_60_00_0
  E: ID_NET_DRIVER=i40e
  E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
  E: ID_NET_NAME=enp96s0f0np0
  E: NM_UNMANAGED=1
  E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp96s0f0np0
  E: TAGS=:systemd:
  E: CURRENT_TAGS=:system:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2085835/+subscriptions