Dell system takes a long time to connect network with external dock

Bug #1843381 reported by Che Cheng on 2019-09-10
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OEM Priority Project
Critical
Che Cheng
systemd (Ubuntu)
Undecided
Unassigned
Bionic
Medium
Dan Streetman
Disco
Medium
Dan Streetman
Eoan
Undecided
Unassigned

Bug Description

[impact]

On Dell system with BIOS-based "MAC passthrough", there can be multiple USB nics with identical MAC addresses. Since the udev rules in Debian and Ubuntu assign interface names for USB nics by mac address (because that is the only consistent identifier for USB nics; their path can change based on which USB port they are connected to), it's impossible to name two interfaces with the same name. As Ubuntu also carries a patch to retry renaming of any interface when the first renaming fails, this causes a 90 second delay before being able to the "MAC passthrough" nic after connecting it.

[test case]

On a system with this "MAC passthrough" enabled and required devices, boot the system and then connect to the dock or connect the second USB nic with identical MAC. It will not be usable for 90 seconds as its renames takes that long to timeout.

[regression potential]

the change here is very limited to only Dell systems with the specific USB vendor/product ID affected by this, and additionally the change only sets a ENV flag in the udev rule, which is later used by udevd to skip the rename-retries for 90 seconds. So, the regression potential for anyone else without a system affected by this "MAC passthrough" should be very low, and any regression potential for those with this "MAC passthrough" should still be low, as this only skips the rename-retry that we know will never succeed.

However, the regression potential is likely limited to failure to properly name a USB nic, or other bugs during the udev processing of new USB nics.

[other info]

original description:
---

This is a bug reopen from
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1837700
The original one caused systemd regressed.
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1842651

This issue needs an alternative solution.
--------------------------------------------------------------------------------
Dell has a feature called MAC addrss passthrough[1] that would force usb ethernet adapters to be assigned with a predefined MAC address stored in BIOS or so. This feature has been landed to mainline kernel in driver r8152[2]. So whenever a r8152 managed device is plugged into Dell devices with MAC addrss passthrough enabled, this driver will set NIC MAC to a predefined one.

And some Dell devices have already one built-in r8152 NIC port. On these devices, when a second r8152 NIC is plugged in, a Debian originated udev rules file 73-usb-net-by-mac.rules[3] will invoke udev built-in command `net_id` to give a persistent name, and that will be based on MAC address. However, since the system has already initialized the built-in r8152 NIC with that name, renaming the second interface with this name will always fail.

While Debian still carries a patch called "Revert-udev-network-device-renaming-immediately-give.patch"[4] that tries to keep support of already deprecated "75-persistent-net-generator.rules" based interface renaming mechanism, this patch also propagated into Ubuntu[5]. This patch will retry renaming with a 90 seconds timeout when the error code is -EEXIST, so the uevent processing will always be blocked in the last ifrename step in the victim system.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: udev 237-3ubuntu10.24 [modified: lib/udev/rules.d/50-firmware.rules lib/udev/rules.d/50-udev-default.rules lib/udev/rules.d/73-special-net-names.rules lib/udev/rules.d/73-usb-net-by-mac.rules]
ProcVersionSignature: Ubuntu 4.15.0-1043.48-oem 4.15.18
Uname: Linux 4.15.0-1043-oem x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
CustomUdevRuleFiles: 70-snap.core.rules 95-oem-hotkey-osd.rules
Date: Wed Jul 24 15:30:59 2019
DistributionChannelDescriptor:
 # This is the distribution channel descriptor for the OEM CDs
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-bionic-amd64-20180608-47+beaver-jorah+X90
InstallationDate: Installed on 2019-07-03 (20 days ago)
InstallationMedia: Ubuntu 18.04 "Bionic" - Build amd64 LIVE Binary 20180608-09:38
MachineType: Dell Inc. Latitude 7424 Rugged Extreme
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1043-oem.efi.signed root=UUID=5da90c85-3500-49a2-b989-71a604f9eec4 ro mem_sleep_default=deep quiet splash systemd.log_level=debug udev.log-priority=debug log_buf_len=8M vt.handoff=1
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/27/2019
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.5.0
dmi.board.name: 0Y7FK3
dmi.board.vendor: Dell Inc.
dmi.board.version: X03
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.5.0:bd05/27/2019:svnDellInc.:pnLatitude7424RuggedExtreme:pvr:rvnDellInc.:rn0Y7FK3:rvrX03:cvnDellInc.:ct10:cvr:
dmi.product.family: Latitude
dmi.product.name: Latitude 7424 Rugged Extreme
dmi.sys.vendor: Dell Inc.

[1]: https://www.dell.com/support/article/tw/zh/twdhs1/sln301147/what-is-mac-address-pass-through?lang=en
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/usb/r8152.c
[3]: https://salsa.debian.org/systemd-team/systemd/blob/master/debian/extra/rules/73-usb-net-by-mac.rules
[4]: https://salsa.debian.org/systemd-team/systemd/blob/master/debian/patches/debian/Revert-udev-network-device-renaming-immediately-give.patch
[5]: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/tree/debian/patches/debian/Revert-udev-network-device-renaming-immediately-give.patch?h=ubuntu-bionic

Che Cheng (cktenn) on 2019-09-10
affects: systemd → oem-priority
Che Cheng (cktenn) on 2019-09-10
Changed in oem-priority:
importance: Undecided → Critical
assignee: nobody → Che Cheng (cktenn)
Che Cheng (cktenn) wrote :

I would like to propose to write a rule to rename a removable Realtek 8153 interface with ID_NET_NAME_PATH instead of ID_NET_NAME_MAC.

The solution may impact users using Realtek 8153 and manipulate it by the interface name since the name is not consistent and will change when switching USB port. Networkmanager is not affected since it identifies interfaces via UUID.

tags: added: id-5d84ab32951ae364f0c9f3b7
Dan Streetman (ddstreet) wrote :

So, am I understanding right that such a system will have two ethernet interfaces with identical mac addresses? Isn't that an obvious problem that should be fixed instead?

On Thu, 26 Sep 2019 at 20:11, Dan Streetman <email address hidden> wrote:
>
> So, am I understanding right that such a system will have two ethernet
> interfaces with identical mac addresses? Isn't that an obvious problem
> that should be fixed instead?

Digging into this, It seems that it is an explicitly supported
configuration that was added in 2016
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1579984

Ie. the dock copies the MAC address of the laptop, such that the same
lease is still usable.

Thus the timeouts now observed, are a regression of said feature.

--
Regards,

Dimitri.

Dan Streetman (ddstreet) wrote :

> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1579984

http://patchwork.ozlabs.org/patch/631759/
"Just resubmit it and I'll apply it, I'm so tired of hearing about this..."

lol I'm relieved to see dmiller and others upstream had exactly the same concerns as me :)

If he decided wasn't bad enough to reject it's good enough for me, although it is still a really bad design.

I think the 73-usb-net-by-mac.rules needs to be changed to account for this weird situation, let's see what Debian thinks.

Che Cheng (cktenn) wrote :

There is 73-special-net-names.rules which 73-usb-net-by-mac.rules was split from, and it'll execute prior to 73-usb-net-by-mac.rules.

We can keep the fixed internal MAC passthrough interface using MAC address as interface name, and let removable dongles using pathname by adding following line to 73-special-net-names.rules.

ACTION=="add", SUBSYSTEM=="net", SUBSYSTEMS=="usb", NAME=="", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", ATTRS{removable}=="removable", IMPORT{builtin}="net_id", NAME="$env{ID_NET_NAME_PATH}"

Rex Tsai (chihchun) on 2019-10-04
tags: added: id-5d84ab32951ae364f0c9f3b7oem-priority
removed: id-5d84ab32951ae364f0c9f3b7
tags: added: id-5d84ab32951ae364f0c9f3b7 oem-priority
removed: id-5d84ab32951ae364f0c9f3b7oem-priority
Dan Streetman (ddstreet) wrote :

@cktenn that could work (using special-net-names), however I'm concerned that the rule should not apply to all r8152 devices, everywhere, on everyone's systems. It should apply *only* on Dell systems where this magical MAC passthrough is enabled. What can be checked to see if the system is a Dell with the special MAC passthrough enabled?

Alternately, if the only problem is 73-usb-net-by-mac.rules fails due to the system having multiple nics with the same mac, that rule could be adjusted to check if an interface with the name already exists, and if so don't try re-setting it (since that would fail). That might be an easier sell to Debian.

What do you think?

Che Cheng (cktenn) wrote :

@Dan

I've submitted a merge request to Debian that only applies to Dell system the idea mentioned in comment #4, it modifies the rule 73-special-net-names.rules.

https://salsa.debian.org/systemd-team/systemd/merge_requests/55

Please help to review this.

Dan Streetman (ddstreet) wrote :

Thanks @cktenn - I commented there and also opened https://salsa.debian.org/systemd-team/systemd/merge_requests/56, but I have not tested with that, can you see if that fixes this problem? Assuming I'm understanding the problem correctly, that you just want 73-special-net-names.rules to ignore the second duplicated-mac interface.

Dan Streetman (ddstreet) wrote :

> Based on https://salsa.debian.org/systemd-team/systemd/merge_requests/56#note_113429,
> is https://salsa.debian.org/systemd-team/systemd/merge_requests/55 suitable for the
> SRU workaround?

Have you actually tested this using an affected system? I'm concerned that changing to use by-path naming will result in a different interface name, for the same adapter, when it's plugged into a different usb port. That doesn't matter for internal (either on-board or in-dock) usb adapters, since their device path never changes, but for people with dell systems and a separate usb nic with this usb vendor/product id, this would be a serious regression if their interface name changes 1) after simply upgrading systemd, and they reboot or hotplug their nic, even if they plug it into the same usb port; and 2) every time they hotplug the usb nic into a different usb port (or use a usb hub, etc.)

Che Cheng (cktenn) wrote :

I would like to explain the original intention of the design in the udev rule.

Since it is not possible letting two NICs to have the same ifname, I tend to find a way to assign a persistent name that would not duplicate on a system for each usbnet. It is meant to break the current rule, so I was trying to reduce the number of potential victims. I found that people using NetworkManager, which identifies connections by UUID, won't be affected. Restrict the system vendor also helps to mitigate the impact.

I admit unsetting the duplicated ifname does solve the original issue without regression in a real case.

Dan Streetman (ddstreet) wrote :

Ok, so just to clarify, the only issue here is the delay/failure for the system to finish bringing its networking up, right? The exact name of the second "duplicated mac" interface doesn't really matter?

Che Cheng (cktenn) wrote :

Yes, since it will be the same result as udev failed to rename it, it doesn't matter.

Dan Streetman (ddstreet) wrote :

@cktenn could you try the systemd package for either bionic and/or disco from here:
https://launchpad.net/~ddstreet/+archive/ubuntu/systemd

That includes:
https://git.launchpad.net/~ubuntu-support-team/ubuntu/+source/systemd/commit/?id=173071ae695e11c1549656a7b40d2201681e733e

I don't love it, but I think that is the 'least bad' way of handling this without affecting any current users.

Changed in systemd (Ubuntu Eoan):
status: New → Invalid
Changed in systemd (Ubuntu Disco):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
Changed in systemd (Ubuntu Disco):
status: New → In Progress
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Dan Streetman (ddstreet) wrote :

Also I marked this invalid for Eoan, as the retry-interface-renaming has been removed in Eoan; @cktenn can you test there and verify this bug doesn't exist for Eoan?

tags: added: bionic ddstreet disco systemd
Che Cheng (cktenn) wrote :

@ddstreet

Tested on bionic/disco/eoan. All of them do not rename the second usbnet with MAC passthrough function, it keeps ifname eth0.

Thank you so much.

Dan Streetman (ddstreet) on 2019-10-18
description: updated

Hello Che, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco

All autopkgtests for the newly accepted systemd (240-6ubuntu5.8) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-bind-exporter/unknown (armhf)
php7.2/7.2.24-0ubuntu0.19.04.1 (armhf)
gvfs/1.40.1-1ubuntu0.1 (ppc64el)
pdns-recursor/unknown (armhf)
webhook/unknown (armhf)
munin/2.0.47-1ubuntu3 (armhf, arm64)
systemd/240-6ubuntu5.8 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Che Cheng (cktenn) wrote :

I can confirm that systemd 240-6ubuntu5.8 fixes this issue for me on disco.

tags: added: verification-done-disco
removed: verification-needed-disco

Hello Che, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.32) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)
linux/unknown (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Che Cheng (cktenn) wrote :

I can confirm that systemd 237-3ubuntu10.32 fixes this issue for me on disco.

tags: added: verification-done-bionic
removed: verification-needed-bionic

Hello Che, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed-bionic
removed: verification-done-bionic
Che Cheng (cktenn) wrote :

I can confirm that systemd 237-3ubuntu10.33 fixes this issue for me on bionic.

tags: added: verification-done-bionic
removed: verification-needed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.33) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (ppc64el, amd64)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
umockdev/0.11.1-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Che Cheng (cktenn) on 2019-11-16
tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers