biosdevname gives name of device as rename7 in Quantal

[Narinder Gupta]

In Quantal kernel biosdevname feature is not working as it suppose to be. On hardware we are seeing an issue where few of the network interface names are named as rename7 etc.. while biosdevname -d utility provides the correct biosname while kernel name have been messed up. There are total 16 network interfaces on the server which are ALOMs for Emulex card connected to system.

It seems this issue is a race condition and will get fixed in Fedora 18. But having fix in Quantal will help also.

https://bugzilla.redhat.com/show_bug.cgi?id=782145

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1090002

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.7 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-raring/

[Narinder Gupta (narindergupta)]

can not run the apport-collect on the machine as machine is not on the network. And proxys are not allowing me to run the same successfully. there is no crash in the package or utility so i don't think apport will collect any useful information from the system.

[Steve Langasek (vorlon)]

Stéphane, you mentioned you were able to reproduce this behavior in a different system with multiple network devices. Can you please follow up on the udev/biosdevname problem?

[Stéphane Graber (stgraber)]

Hi,

Can you please attach:
 - /var/log/syslog
 - /var/log/udev
 - output of "dmesg"
 - output of "ifconfig -a"
 - /etc/udev/rules.d/70-persistent-net.rules

That last file may be the source of some of your problems. Once you're done attaching them all, can you try to comment all the entries in /etc/udev/rules.d/70-persistent-net.rules and see if that helps?

[Narinder Gupta (narindergupta)]

I am attaching the logs you also. also i am attaching the logs of biosdevname -d which will give you info that biosname of device is correct. I am seeing 70-persistent-net.rules is empty and rules to generate the name is in 71-biosdevname.rules which i am attaching. After commenting the line in 71-biosdevname.rules which generates the name en<1-16> and rename<> i am seeing all devices are named as eth<0-17> and 70-persistent-net.rules contain the mapping of MAC address with the eth<0-15> device name.

I am attaching the below required logs.
 - /var/log/syslog
 - /var/log/udev
 - output of "dmesg"
 - output of "ifconfig -a"
 - /etc/udev/rules.d/70-persistent-net.rules
- /lib/udev/rules.d/71-biosdevname.rules
- biosdevname -d

[Narinder Gupta (narindergupta)]

adding debug.tar which we got after running few biosdevname command during udev.

[Stéphane Graber (stgraber)]

Ok, so I've been doing a big of debugging with the help of Narinder.

One thing that I found really quite odd is that the BIOS name to MAC address mapping isn't static as you'd think.
Instead it appears to change several during the boot sequence.

The list below shows the BIOS name on the left and the MAC address on the right. This was generated by calling biosdevname -d every time an interface appeared:

em1 => 00:17:A4:77:3C:00

em2 => 00:17:A4:77:3C:02
em2 => 00:17:A4:77:3C:08

em3 => 00:17:A4:77:3C:08
em3 => 00:17:A4:77:3C:0A

em4 => 00:17:A4:77:3C:0A
em4 => B4:99:BA:A6:B9:18
em4 => B4:99:BA:A6:B9:19
em4 => B4:99:BA:A6:B9:1D

em5 => 00:17:A4:77:3C:0C
em5 => B4:99:BA:A6:B9:18
em5 => B4:99:BA:A6:B9:19
em5 => B4:99:BA:A6:B9:1C
em5 => B4:99:BA:A6:B9:1D

em6 => 00:17:A4:77:3C:06
em6 => B4:99:BA:A6:B9:19
em6 => B4:99:BA:A6:B9:1C
em6 => B4:99:BA:A6:B9:1D
em6 => B4:99:BA:A6:B9:20
em6 => B4:99:BA:A6:B9:24

em7 => 00:17:A4:77:3C:06
em7 => 00:17:A4:77:3C:0C
em7 => B4:99:BA:A6:B9:19
em7 => B4:99:BA:A6:B9:1D
em7 => B4:99:BA:A6:B9:24
em7 => B4:99:BA:A6:B9:25

em8 => 00:17:A4:77:3C:04
em8 => 00:17:A4:77:3C:0C
em8 => B4:99:BA:A6:B9:1D
em8 => B4:99:BA:A6:B9:20

em9 => 00:17:A4:77:3C:04
em9 => 00:17:A4:77:3C:06
em9 => B4:99:BA:A6:B9:20
em9 => B4:99:BA:A6:B9:24

em10 => 00:17:A4:77:3C:06
em10 => 00:17:A4:77:3C:0C
em10 => B4:99:BA:A6:B9:24
em10 => B4:99:BA:A6:B9:25

em11 => 00:17:A4:77:3C:0C
em11 => 00:17:A4:77:3C:0E
em11 => B4:99:BA:A6:B9:25

em12 => 00:17:A4:77:3C:0E
em12 => B4:99:BA:A6:B9:20

em13 => B4:99:BA:A6:B9:20
em13 => B4:99:BA:A6:B9:21
em13 => B4:99:BA:A6:B9:24

em14 => B4:99:BA:A6:B9:21
em14 => B4:99:BA:A6:B9:24
em14 => B4:99:BA:A6:B9:25

em15 => B4:99:BA:A6:B9:21
em15 => B4:99:BA:A6:B9:25

em16 => B4:99:BA:A6:B9:25

[Stéphane Graber (stgraber)]

You can see the same phenomenon in the tarball that Narinder posted above.

Here's the em3 definition at the time em1 appeared:
BIOS device: em3
Kernel name: em3
Permanent MAC: 00:17:A4:77:3C:08
Assigned MAC : 00:17:A4:77:3C:08
ifIndex: 4
Driver: be2net
Driver version: 4.2.220u
Firmware version: 4.1.450.7
Bus Info: 0000:04:00.2
PCI name : 0000:04:00.2
PCI Slot : embedded
Embedded Index: 3

And then at the time em2 appeared:
BIOS device: em3
Kernel name: em5
Permanent MAC: B4:99:BA:A6:B9:18
Assigned MAC : B4:99:BA:A6:B9:18
ifIndex: 6
Driver: be2net
Driver version: 4.2.220u
Firmware version: 4.1.450.7
Bus Info: 0000:04:00.4
PCI name : 0000:04:00.4
PCI Slot : embedded
SMBIOS Device Type: Ethernet
SMBIOS Instance: 5
Embedded Index: 5

As you can see above, the same BIOS entry "em3" switched mac address and now refers to a completely different NIC.

[Narinder Gupta (narindergupta)]

debug logs for lspci, dmidecode and biosdecode have been attached.

[jordanh (jordan-hargrave)]

It looks like those NICs getting renumbered are setup as iSCSI NICs.. can you disable that feature and see if it works properly? Are you running a VM and remapping/claiming those PCI devices elsewhere?

[Narinder Gupta (narindergupta)]

Both controller ports are NIC types. But i can see the same MAC address for both NIC iSCSI type of device and it may cause the biosdevname not to act correctly. Here are the NICs MAC address exposed to the system. Only option i have here in BIOS is either iSCSI or FCOE for four embedded ports. I am contacting with NIC team to find out is there any way to overcome having same MAC for different type of NIC and iSCSI.

NIC Port 1 00:17:a4:77:3c:00
NIC Port 2 00:17:a4:77:3c:02
NIC Port 9 00:17:a4:77:3c:04
NIC Port 10 00:17:a4:77:3c:06
NIC Port 3 00:17:a4:77:3c:08
iSCSI Port 1 00:17:a4:77:3c:08
NIC Port 4 00:17:a4:77:3c:0a
iSCSI Port 2 00:17:a4:77:3c:0a
NIC Port 11 00:17:a4:77:3c:0c
iSCSI Port 3 00:17:a4:77:3c:0c
NIC Port 12 00:17:a4:77:3c:0e
iSCSI Port 4 00:17:a4:77:3c:0e
NIC Port 5 b4:99:ba:a6:b9:18
NIC Port 7 b4:99:ba:a6:b9:19
NIC Port 6 b4:99:ba:a6:b9:1c
NIC Port 8 b4:99:ba:a6:b9:1d
NIC Port 13 b4:99:ba:a6:b9:20
NIC Port 15 b4:99:ba:a6:b9:21
NIC Port 14 b4:99:ba:a6:b9:24
NIC Port 16 b4:99:ba:a6:b9:25

[jordanh (jordan-hargrave)]

I think this has something to do with the HP SMBIOS entries.

Incidentally it looks like there is a BIOS bug as Type 9 Slot structures all point to 0000:00:00.0 (an actual device) not ffff:ff:1f.7 for an inactive device.

Type 209 lists:
 NIC 1: PCI device 04:00.0, MAC address 00:17:A4:77:3C:00
 NIC 2: PCI device 04:00.1, MAC address 00:17:A4:77:3C:02
 NIC 3: PCI device 04:00.2, MAC address 00:17:A4:77:3C:08
 NIC 4: PCI device 04:00.3, MAC address 00:17:A4:77:3C:0A
 NIC 5: PCI device 04:00.4, MAC address B4:99:BA:A6:B9:18
 NIC 6: PCI device 04:00.5, MAC address B4:99:BA:A6:B9:1C
 NIC 7: PCI device 04:00.6, MAC address B4:99:BA:A6:B9:19
 NIC 8: PCI device 04:00.7, MAC address B4:99:BA:A6:B9:1D
 NIC 9: PCI device 05:00.0, MAC address 00:17:A4:77:3C:04
 NIC 10: PCI device 05:00.1, MAC address 00:17:A4:77:3C:06
 NIC 11: PCI device 05:00.2, MAC address 00:17:A4:77:3C:0C
 NIC 12: PCI device 05:00.3, MAC address 00:17:A4:77:3C:0E
 NIC 13: PCI device 05:00.4, MAC address B4:99:BA:A6:B9:20
 NIC 14: PCI device 05:00.5, MAC address B4:99:BA:A6:B9:24
 NIC 15: PCI device 05:00.6, MAC address B4:99:BA:A6:B9:21
 NIC 16: PCI device 05:00.7, MAC address B4:99:BA:A6:B9:25

type 221 lists:
 NIC 1: PCI device 04:00.2, MAC address 00:17:A4:77:3C:08
 NIC 2: PCI device 04:00.3, MAC address 00:17:A4:77:3C:0A
 NIC 3: PCI device 05:00.2, MAC address 00:17:A4:77:3C:0C
 NIC 4: PCI device 05:00.3, MAC address 00:17:A4:77:3C:0E

smbios is using both 209/221 so it is assigning index
3/1 to 00:17:a4:77:3c:08
4/2 to 00:17:a4:77:3c:0a
11/3 to 00:17:a4:77:3c:0c
12/4 to 00:17:a4:77:3c:0e

Not sure why the others are getting renumbered in the process though

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul Boven (p-boven)]

Same problem on two identical machines that have dual 1Gb/s ethernet cards on the motherboard.
This apparently causes booting to stall for about a minute, see this snippet from syslog after a boot. These timestamps are
while the machine is booting, and hasn't gotten to the point where it presents a login prompt.

Sep 27 15:48:31 cl1 ntpd[1432]: new interface(s) found: waking up resolver
Sep 27 15:49:54 cl1 udevd[691]: error changing net interface name rename3 to p2p1: File exists

That interface really shouldn't be renamed to p2p1 though!

BIOS device: p2p2
Kernel name: rename3
Permanent MAC: 00:25:90:64:6D:E9
Assigned MAC : 00:25:90:64:6D:E9
ifIndex: 3
Driver: igb
Driver version: 4.1.2-k
Firmware version: 1.96, 0x8000090e
Bus Info: 0000:02:00.1
PCI name : 0000:02:00.1
PCI Slot : 2
Index in slot: 2

Machines are running Raring Ringtail 3.8.0.31 (patches updated today, didn't have this issue previously)

[Narinder Gupta (narindergupta)]

On 09/27/2013 09:18 AM, Paul Boven wrote:
> Same problem on two identical machines that have dual 1Gb/s ethernet cards on the motherboard.
> This apparently causes booting to stall for about a minute, see this snippet from syslog after a boot. These timestamps are
> while the machine is booting, and hasn't gotten to the point where it presents a login prompt.
>
> Sep 27 15:48:31 cl1 ntpd[1432]: new interface(s) found: waking up resolver
> Sep 27 15:49:54 cl1 udevd[691]: error changing net interface name rename3 to p2p1: File exists
>
> That interface really shouldn't be renamed to p2p1 though!
name of the controller in OS should be p2p1. As it seems this controller
is in slot 2 and port 2.
>
> BIOS device: p2p2
> Kernel name: rename3
> Permanent MAC: 00:25:90:64:6D:E9
> Assigned MAC : 00:25:90:64:6D:E9
> ifIndex: 3
> Driver: igb
> Driver version: 4.1.2-k
> Firmware version: 1.96, 0x8000090e
> Bus Info: 0000:02:00.1
> PCI name : 0000:02:00.1
> PCI Slot : 2
> Index in slot: 2
>
> Machines are running Raring Ringtail 3.8.0.31 (patches updated today,
> didn't have this issue previously)
>

--
Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Technical Account Manager
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

[Paul Boven (p-boven)]

Hi Narinder,

Well, that's exactly the problem - sometime during the boot, apparently biosdev thought it was p2p1, and the OS tried to assign the name p2p1 to it, which it had already given out. When I run biosdev now, it shows up as being p2p2 allright - but the interface doesn't get assigned the name p2p2 during boot anymore, and instead I have an unexpected interface name 'rename3' show up in my ifconfig. Which breaks things quite badly.

[Paul Boven (p-boven)]

Ugh, I also just noticed that this messes up the order of my interfaces in SNMP - it's swapped the two ones, so now my graphs that used to show the external interfaces, are showing the internal ones, and v.v.

[Narinder Gupta (narindergupta)]

Yeah we ran this issue with biosdevname developer and after close
examination he told thats the issue with the BIOS firmware where SMBIOS
entry were not proper. When we tried to run this on later HP FW system
we do not see the bug anymore.

Will you please let me know you server configuraiton include the
firmware version?

On 09/27/2013 10:43 AM, Paul Boven wrote:
> Hi Narinder,
>
> Well, that's exactly the problem - sometime during the boot, apparently
> biosdev thought it was p2p1, and the OS tried to assign the name p2p1 to
> it, which it had already given out. When I run biosdev now, it shows up
> as being p2p2 allright - but the interface doesn't get assigned the name
> p2p2 during boot anymore, and instead I have an unexpected interface
> name 'rename3' show up in my ifconfig. Which breaks things quite badly.
>

--
Thanks and Regards,
Narinder Gupta (PMP) <email address hidden>
Technical Account Manager
Canonical, Ltd. narindergupta [irc.freenode.net]
+1.281.736.5150 narindergupta2007[skype]

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

[Paul Boven (p-boven)]

Server configuration:

SuperMicro X9DRi-F mainboard
On-board dual I350 (rev 01) Gigabit Ethernet controller [8086:1521], igb driver
PCI-E X540-AT2 (rev01) 10Gbase-T Ethernet card [8086:1528], ixgbe driver.

bios: Version 2.0a, 03/27/2013

The current situation causes two issues:
1.) The former interface p2p2 is now consistently named 'rename3'.
2.) There's a delay of more than a minute during the boot process, where nothing happens.

[Mike Solomon (msolo)]

I'm seeing the same behavior in 13.10 saucy with some Chelsio cards. They consistently show up as rename3 and rename4.

Was there any resolution to your situation?

I can supply my logs as well if that would help.