Thank you for your quick response.

We are very interested in solving this issue as this servers are part of our critical infrastructure - mongodb replica set, so I secured a permission to restart the servers.

Here are the attached udev.log files. One is the abnormal where we had the issue and the normal is from one of our other machines unaffected by the issue.
I have zipped the files as they were quite large ~6MB

________________________________
From: <email address hidden> <email address hidden> on behalf of Nick Rosbrook <email address hidden>
Sent: Tuesday, October 29, 2024 15:28
To: Stoyan Atanasov <email address hidden>
Subject: [Bug 2085835] Re: "Predictable Network Interface Naming" suddenly changed the name from enp96s0f0 to enp96s0f0np0 causing outage

By default (and as we can see in the ID_NET_LINK_FILE= property above),
the name policy is controlled by NamePolicy= in
/usr/lib/systemd/network/99-default.link. On Jammy, this looks like:

root@jammy:~# cat /lib/systemd/network/99-default.link
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

Basically, udev will try those policies in order until one matches.
Based on the udev properties above, it looks like the 'path' policy was
selected. This *might* different than before, or maybe something else
changed on the system that then changed resulting name constructed by
udev.

Can you please enable debug logs for udev (ideally on both a working and
non-working system), and share them here? E.g.

$ mkdir -p /etc/systemd/system/systemd-udevd.service.d
$ cat  > /etc/systemd/system/systemd-udevd.service.d/debug.conf << EOF
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
EOF

Then, reboot, and then run:

$ journalctl -u systemd-udevd.service -b > udev.log

and attach the result.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/2085835

Title:
  "Predictable Network Interface Naming" suddenly changed the name from
  enp96s0f0 to enp96s0f0np0 causing outage

Status in systemd package in Ubuntu:
  New

Bug description:
  On three identical servers - sleds in a blade chassis.
  the NIC name of one of the servers suddenly changed, causing an outage. The netplan config could not assign IP to the interface due to the name change.

  enp96s0f0 to enp96s0f0np0

  The two other servers still have the enp96s0f0 name. No BIOS or other
  configuration change is detectable between the three systems.
  Firmware, software - all the same


  I tried to debug this issue myself, but it turned out to be a very niche and complicated topic.
  I went to submit an issue on https://github.com/systemd/systemd/issues, but they only accept issues for newer versions.

  System info:

  dundts@mongodb-rs2:~$ lsb_release -rd
  Description:    Ubuntu 22.04.3 LTS
  Release:        22.04

  dundts@mongodb-rs2:~$ apt-cache policy systemd
  system:
    Installiert:           249.11-0ubuntu3.11
    Installationskandidat: 249.11-0ubuntu3.12
    Versionstabelle:
       249.11-0ubuntu3.12 500
          500 http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
   *** 249.11-0ubuntu3.11 100
          100 /var/lib/dpkg/status
       249.11-0ubuntu3.7 500
          500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
       249.11-0ubuntu3 500
          500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

  dundts@mongodb-rs2:~$ sudo dmesg | grep i40e
  [sudo] password for dundts:
  [    3.657531] i40e: Intel(R) Ethernet Connection XL710 Network Driver
  [    3.658208] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
  [    3.715866] i40e 0000:60:00.0: fw 3.1.54559 api 1.5 nvm 3.2d 0x80000b4b 1.1767.0 [8086:37d3] [152d:8a40]
  [    3.720222] i40e 0000:60:00.0: MAC address: d8:c4:97:4c:66:ae
  [    3.720558] i40e 0000:60:00.0: FW LLDP is enabled
  [    3.732613] i40e 0000:60:00.0: Added LAN device PF0 bus=0x60 dev=0x00 func=0x00
  [    3.733075] i40e 0000:60:00.0: Features: PF-id[0] VFs: 32 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [    3.788671] i40e 0000:60:00.1: fw 3.1.54559 api 1.5 nvm 3.2d 0x80000b4b 1.1767.0 [8086:37d3] [152d:8a40]
  [    3.804478] i40e 0000:60:00.1: MAC address: d8:c4:97:4c:66:af
  [    3.804964] i40e 0000:60:00.1: FW LLDP is enabled
  [    3.839982] i40e 0000:60:00.1 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
  [    3.851939] i40e 0000:60:00.1: Added LAN device PF1 bus=0x60 dev=0x00 func=0x01
  [    3.877994] i40e 0000:60:00.1: Features: PF-id[1] VFs: 32 VSIs: 66 QP: 32 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [    4.065330] i40e 0000:60:00.0 enp96s0f0np0: renamed from eth0
  [    4.096320] i40e 0000:60:00.1 enp96s0f1np1: renamed from eth1

  dundts@mongodb-rs2:~$ sudo udevadm info /sys/class/net/enp96s0f0np0
  P: /devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/0000:5f:03.0/0000:60:00.0/net/enp96s0f0np0
  L: 0
  E: DEVPATH=/devices/pci0000:5d/0000:5d:00.0/0000:5e:00.0/0000:5f:03.0/0000:60:00.0/net/enp96s0f0np0
  E: INTERFACE=enp96s0f0np0
  E: IFINDEX=2
  E: SUBSYSTEM=net
  E: USEC_INITIALIZED=4079509
  E: ID_MM_CANDIDATE=1
  E: ID_NET_NAMING_SCHEME=v249
  E: ID_NET_NAME_MAC=enxd8c4974c66ae
  E: ID_OUI_FROM_DATABASE=Quanta Computer Inc.
  E: ID_NET_NAME_PATH=enp96s0f0np0
  E: ID_BUS=pci
  E: ID_VENDOR_ID=0x8086
  E: ID_MODEL_ID=0x37d3
  E: ID_PCI_CLASS_FROM_DATABASE=Network controller
  E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
  E: ID_VENDOR_FROM_DATABASE=Intel Corporation
  E: ID_MODEL_FROM_DATABASE=Ethernet Connection X722 for 10GbE SFP+
  E: ID_PATH=pci-0000:60:00.0
  E: ID_PATH_TAG=pci-0000_60_00_0
  E: ID_NET_DRIVER=i40e
  E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
  E: ID_NET_NAME=enp96s0f0np0
  E: NM_UNMANAGED=1
  E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enp96s0f0np0
  E: TAGS=:systemd:
  E: CURRENT_TAGS=:system:

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2085835/+subscriptions