systemd 229-4ubuntu6 ignores net.ifnames=0 on USB or /etc/udev/rules.d/80-net-setup-link.rules being a /dev/null symlink

[Steve Wormley]

Description: Ubuntu 16.04 LTS
Release: 16.04

The upgrade to systemd/udev 229-4ubuntu6 breaks net.ifnames=0 for USB devices.

It appears the regression is here:
* Set MAC based name for USB network interfaces only for universally
    administered (i. e. stable) MACs, not for locally administered (i. e.
    randomly generated) ones. Drop /lib/systemd/network/90-mac-for-usb.link
    (as link files don't currently support globs for MACAddress=) and replace
    with an udev rule in /lib/udev/rules.d/73-special-net-names.rules.
    (Closes: #812575, LP: #1574483)

As Raspberry Pi's use eth0 via USB, this breaks running systems.

Before:
ii systemd 229-4ubuntu4 armhf system and service manager
ii udev 229-4ubuntu4 armhf /dev/ and hotplug management daem

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether b8:27:eb:16:39:e9 brd ff:ff:ff:ff:ff:ff

After:

ii systemd 229-4ubuntu6 armhf system and service manager
ii udev 229-4ubuntu6 armhf /dev/ and hotplug management daemon

3: enxb827eb1639e9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether b8:27:eb:16:39:e9 brd ff:ff:ff:ff:ff:ff

cat /proc/cmdline
dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2709.boardrev=0xa01041 bcm2709.serial=0x37b38253 smsc95xx.macaddr=B8:27:EB:B3:82:53 bcm2708_fb.fbswap=1 bcm2709.disk_led_gpio=47 bcm2709.disk_led_active_low=0 sdhci-bcm2708.emmc_clock_freq=250000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

With the default interfaces configuration, all networking is lost on reboot after upgrade.

SRU TEST CASE
=============
 * Boot with "net.ifnames=0" on the kernel command line, and connect an USB ethernet device. It will still be called enxDEADBEEF with current xenial. With the -proposed version it will instead keep the kernel name, like "usb0" as intended.

 * Do "sudo ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules" (the other documented way to disable ifnames) and do the above connect/name check test again.

[Steve Wormley (9-lp)]

Yes, /proc/cmdline.txt was from a different system thus the differing MAC addresses.

[Martin Pitt (pitti)]

This was fixed in http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=b3e7c6ee, thus is fixed in yakkety. I'll backport the fix to xenial.

[Martin Pitt (pitti)]

Actually we are still missing checking /etc/udev/rules.d/80-net-setup-link.rules → /dev/null, the other documented way how to disable ifnames.

[Martin Pitt (pitti)]

Fixed second half in http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=93e26e

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 230-5

---------------
systemd (230-5) unstable; urgency=medium

  [ Martin Pitt ]
  * Sync test/networkd-test.py with current upstream master, and remove our
    debian/tests/networkd copy. Directly run test/networkd-test.py in
    autopkgtest.
  * debian/extra/rules/73-usb-net-by-mac.rules: Disable when
    /etc/udev/rules.d/80-net-setup-link.rules is a symlink to /dev/null, to be
    consistent with the documented way to disable ifnames. (Closes: #824491,
    LP: #1593379)
  * debian/rules: Ignore libcap-ng.so in the "does anything link against /usr"
    check, to work around libaudit1 recently gaining a new dependency against
    that library (#828991). We have no influence on that ourselves. This fixes
    the FTBFS in the meantime.

  [ Felipe Sateler ]
  * Convert common code into a private shared library. This saves about 9 MB
    of installed size in the systemd package, and some more in systemd-*.

 -- Martin Pitt <email address hidden> Fri, 01 Jul 2016 09:15:12 +0200

[Chris J Arges (arges)]

Hello Steve, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

I installed the -proposed binaries on Sebastien's laptop. Disabling ifnames with /etc/udev/rules.d/80-net-setup-link works fine, but disabling it with net.ifnames=0 doesn't. It does in yakkety with the same udev rule, but apparently there's something wrong with IMPORT{cmdline}="net.ifnames" in the xenial version.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu7

---------------
systemd (229-4ubuntu7) xenial-proposed; urgency=medium

  * Add pre-dependency to dpkg >= 1.17.14 on udev, to ensure that
    dpkg-maintscript-helper symlink_to_dir is available. (LP: #1585406)
  * Add activation rate limiting for socket units. (LP: #1568094)
  * Split out udev rule to name USB network interfaces by MAC address into
    73-usb-net-by-mac.rules, so that it's easier to disable. (Closes: #824025)
  * 73-usb-net-by-mac.rules: Disable when net.ifnames=0 is specified on the
    kernel command line or if /etc/udev/rules.d/80-net-setup-link.rules is a
    symlink to /dev/null, to be consistent with disabling the *.link files and
    the documented way to disable ifnames. (Closes: #824491, LP: #1593379)
  * coredump: Fix "Coredump file descriptor missing". (LP: #1602256)

 -- Martin Pitt <email address hidden> Tue, 12 Jul 2016 17:37:25 +0200

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Marek Dopiera (3z-marek-2q)]

I might be doing something broken, but it still seems broken to me on Ubuntu 16.04 on Raspberry PI despite having these fixes applied.

Package versions:
systemd: 229-4ubuntu7
udev: 229-4ubuntu7

Could you please take a look if I'm not doing something stupid (I hopefully attached the relevant info):

$ ifconfig -a | head
enxb827eb739cc2 Link encap:Ethernet HWaddr b8:27:eb:73:9c:c2
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host

$ ls -l /etc/udev/rules.d/80-net-setup-link.rules | tee -a report.txt
lrwxrwxrwx 1 root root 9 Aug 4 2016 /etc/udev/rules.d/80-net-setup-link.rules -> /dev/null

$ grep net.ifnames=0 /proc/cmdline | tee -a report.txt
8250.nr_uarts=1 dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2709.boardrev=0xa02082 bcm2709.serial=0xb2739cc2 smsc95xx.macaddr=B8:27:EB:73:9C:C2 bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 fsck.mode=force net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

$ cat /lib/udev/rules.d/73-usb-net-by-mac.rules
# Use MAC based names for network interfaces which are directly or indirectly
# on USB and have an universally administered (stable) MAC address (second bit
# is 0).

IMPORT{cmdline}="net.ifnames", ENV{net.ifnames}=="0", GOTO="usb_net_by_mac_end"
PROGRAM="/bin/readlink /etc/udev/rules.d/80-net-setup-link.rules", RESULT=="/dev/null", GOTO="usb_net_by_mac_end"

ACTION=="add", SUBSYSTEM=="net", SUBSYSTEMS=="usb", NAME=="", \
    ATTR{address}=="?[014589cd]:*", \
    IMPORT{builtin}="net_id", NAME="$env{ID_NET_NAME_MAC}"

[hackeron (hackeron)]

Any solution to this? - I'm passing net.ifnames=0 biosdevname=0 but after the latest apt-get upgrade, it is ignored :( --

deployer@ubuntu:~$ uname -a
Linux ubuntu 4.4.0-1021-raspi2 #27-Ubuntu SMP Fri Aug 12 11:44:06 UTC 2016 armv7l armv7l armv7l GNU/Linux

deployer@ubuntu:~$ cat /proc/cmdline
8250.nr_uarts=1 dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2709.boardrev=0xa02082 bcm2709.serial=0x80f5f71c smsc95xx.macaddr=B8:27:EB:F5:F7:1C bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait net.ifnames=0 biosdevname=0

deployer@ubuntu:~$ ifconfig -a
enxb827ebf5f71c Link encap:Ethernet HWaddr b8:27:eb:f5:f7:1c
          inet addr:192.168.101.69 Bcast:192.168.101.255 Mask:255.255.255.0
          inet6 addr: fe80::ba27:ebff:fef5:f71c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:666 errors:0 dropped:0 overruns:0 frame:0
          TX packets:514 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:106425 (106.4 KB) TX bytes:94870 (94.8 KB)

[Julia Palandri (julia-palandri)]

I tried with this solution in a raspberry pi2 and it didn't work for me (as discussed with pitti), but adding the new strange iface name to /etc/network/interfaces worked, with what used to be eth0.

On the other hand, the symlinking worked to have recognized a new wireless usb interface, which right after the upgrade that deleted my interfaces had the new weird interface format (ex enxb827ebf5f71c), and then became wlan0.

[Martin Pitt (pitti)]

Unsetting "in progress", as currently I have no idea why the ENV{net.ifnames} match does not work on armhf. This needs to be debugged on a machine that reproduces this, and it does work on mine, in QEMU, in our clouds etc. So if anyone can provide me ssh access to an affected machine, I'm happy to debug it there. I need systemd's build dependencies, but not necessarily root. My key is https://launchpad.net/~pitti/+sshkeys .

[alp (atoker)]

The problem appears to be down to a syntax error - here's the fix we're using as a patch against current systemd in Xenial:

Splitting the IMPORT and ENV/GOTO directives onto separate lines gets things working.

With the patch deployed, the kernel parameter works as specified, allowing us to access hosts remotely after boot. We've tested the patch both with and without net.ifnames boot parameter and the USB ethernet device is assigned correctly as expected for each mode.

This should fix things for Raspberry Pi users as well..

--- /lib/udev/rules.d/73-usb-net-by-mac.rules.orig 2016-09-14 20:40:11.269999878 +0000
+++ /lib/udev/rules.d/73-usb-net-by-mac.rules 2016-09-14 20:42:05.719999834 +0000
@@ -3,7 +3,8 @@
 # is 0). Don't do this when ifnames is disabled via kernel command line or
 # customizing/disabling 80-net-setup-link.rules.

-IMPORT{cmdline}="net.ifnames", ENV{net.ifnames}=="0", GOTO="usb_net_by_mac_end"
+IMPORT{cmdline}="net.ifnames"
+ENV{net.ifnames}=="0", GOTO="usb_net_by_mac_end"

 ACTION=="add", SUBSYSTEM=="net", SUBSYSTEMS=="usb", NAME=="", \
     ATTR{address}=="?[014589cd]:*", \

[alp (atoker)]

Note that, contrary to the original bug report, this isn't an armhf issue but affects all platforms.

We used an external USB ethernet adapter to verify that that fix outlined in #14 also resolves the same issue on Xenial Desktop AMD64.

[Martin Pitt (pitti)]

Interesting observation, thanks! I cannot confirm that the single line is generally broken -- I've tested it on two different computers (although only amd64) and it works fine. However, there is no harm in splitting the rule, so I'll do that for the next SRU if it helps things.

[Martin Pitt (pitti)]

Committed to master branch: https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=72a90870

Cherry-picked into xenial branch for next SRU round: https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-xenial&id=f79508898

[Andy Whitcroft (apw)]

Hello Steve, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

Current SRU got shadowed by a security update, resetting. Will reupload shortly.

[Chris Halse Rogers (raof)]

Hello Steve, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[hackeron (hackeron)]

I'm on Xenial and I ran an apt-get update; apt-get dist-upgrade and my eth0 changed back to enxb827djdk after reboot :(

My biggest issue is virtual interfaces do NOT work with these "predictable" network names. I can do ifconfig eth0:0 192.168.123.123 but I cannot do ifconfig enxb827djdk:0 192.168.123.123 - it just overrites the main IP, rather than creating another virtual interface :(

Previously, I was able to follow these steps:

Edit/boot/firmware/config.txt to add:
  net.ifnames=0
  biosdevname=0

Run: ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules

Make sure this file is absent (which it was anyway): /lib/udev/rules.d/73-usb-net-by-mac.rules

Run: update-initramfs -k all -u

But now, running the above steps no longer works and I'm getting enxb827djdk again :(

I also checked /proc/cmdline and I have net.ifnames=0 there so it's definitely being passed. I'm running kernel 4.4.0-1023-raspi2 Sep 6 16:12:44 UTC 2016

Is there a solution/workaround?

[Martin Pitt (pitti)]

hackeron, which systemd version are you using? Note that the one currently in xenial-updates reverted the previous SRU, you need version 229-4ubuntu11.

[Robie Basak (racb)]

On Fri, Oct 07, 2016 at 03:20:29PM -0000, hackeron wrote:
> My biggest issue is virtual interfaces do NOT work with these
> "predictable" network names. I can do ifconfig eth0:0 192.168.123.123
> but I cannot do ifconfig enxb827djdk:0 192.168.123.123 - it just
> overrites the main IP, rather than creating another virtual interface :(

I can't help with the bug in general, but for this particular use case,
try "ip addr add 192.168.123.123 dev enxb827djdk". ifconfig isn't going
away, but all new development has moved to the "ip" command which
replaces it.

[hackeron (hackeron)]

I'm on systemd 229-4ubuntu10

[Martin Pitt (pitti)]

Right. Can you please install the version from -proposed (see comment #20) and check if that fixes it, i. e. net.ifnames=0 then gets respected?

[Martin Pitt (pitti)]

I tested this on a ThinkPad (amd64) and it works. I cannot reproduce the previously reported failure though, i. e. the one-line udev rule works for me as well. But at least there is no regression here, so IMHO it's safe to release that to get the other fixes out.

If it still does not work for you, please reopen and yell here. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu11

---------------
systemd (229-4ubuntu11) xenial; urgency=medium

  * 73-usb-net-by-mac.rules: Split kernel command line import line.
    Reportedly this makes the rule actually work on some platforms. Thanks
    Alp Toker! (LP: #1593379)
  * fsckd: Do not exit on idle timeout if there are still clients connected
    (Closes: #788050, LP: #1547844)
  * libnss-*.prerm: Remove possible [key=value] options from NSS modules as
    well. (LP: #1625584)
  * Backport networkd 231. Compared to 229 this has a lot of fixes, some of
    which we need for good netplan support. Backporting them individually
    would be a lot more work and a lot less robust, and we did not use/support
    networkd in 16.04 so far. Drop the other network related patches as they
    are included in this backport now. (LP: #1627641)
  * debian/tests/networkd: Re-enable the the DHCPv6 tests. The DHCPv6
    behaviour is fixed with the above backport now.
  * pid1: process zero-length notification messages again. Just remove the
    assertion, the "n" value was not used anyway. This fixes a local DoS due
    to unprocessed/unclosed fds which got introduced by the previous fix.
    (LP: #1628687)
  * pid1: Robustify manager_dispatch_notify_fd(). If
    manager_dispatch_notify_fd() fails and returns an error then the handling
    of service notifications will be disabled entirely leading to a
    compromised system. (side issue of LP: #1628687)

 -- Martin Pitt <email address hidden> Tue, 04 Oct 2016 21:43:04 +0200