ifup of vlan interfaces failing during networking start - RTNETLINK answers: File exists

[James Page]

/e/n/i:

auto lo
iface lo inet loopback
    dns-nameservers 10.245.168.2
    dns-search dellstack
auto eth0
iface eth0 inet static
    gateway 10.245.168.1
    address 10.245.168.17/21
    dns-nameservers 10.245.168.2
    mtu 1500

auto eth1
iface eth1 inet manual
    mtu 1500

auto eth1.2667
iface eth1.2667 inet static
    address 10.245.184.20/24
    vlan-raw-device eth1
    mtu 9000
    vlan_id 2667

output from networking startup:

● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           └─50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Mon 2016-04-04 12:14:26 UTC; 1h 33min ago
     Docs: man:interfaces(5)
  Process: 1255 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 868 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited,
 Main PID: 1255 (code=exited, status=1/FAILURE)

Apr 04 12:14:25 reflecting-attraction systemd[1]: Starting Raise network interfaces...
Apr 04 12:14:26 reflecting-attraction ifup[1255]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Apr 04 12:14:26 reflecting-attraction ifup[1255]: RTNETLINK answers: File exists
Apr 04 12:14:26 reflecting-attraction ifup[1255]: Failed to bring up eth1.2667.
Apr 04 12:14:26 reflecting-attraction systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 12:14:26 reflecting-attraction systemd[1]: Failed to start Raise network interfaces.
Apr 04 12:14:26 reflecting-attraction systemd[1]: networking.service: Unit entered failed state.
Apr 04 12:14:26 reflecting-attraction systemd[1]: networking.service: Failed with result 'exit-code'.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: ifupdown 0.8.10ubuntu1
ProcVersionSignature: User Name 4.4.0-16.32-generic 4.4.6
Uname: Linux 4.4.0-16-generic x86_64
ApportVersion: 2.20.1-0ubuntu1
Architecture: amd64
Date: Mon Apr 4 13:42:48 2016
SourcePackage: ifupdown
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

Turned out it was the mtu setting directly on the vlan interface that causes the problem:

auto eth1.2667
iface eth1.2667 inet static
    address 10.245.184.20/24
    vlan-raw-device eth1
    mtu 9000
    vlan_id 2667

mtu needs to be set on the underlying network device (eth1 in this case).

[James Page (james-page)]

correction - the mismatch between the underlying network device and the vlan interface causes the error:

Apr 04 14:15:23 reflecting-attraction ifup[5936]: run-parts: executing /etc/network/if-pre-up.d/vlan
Apr 04 14:15:23 reflecting-attraction ifup[5936]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Apr 04 14:15:23 reflecting-attraction ifup[5936]: Added VLAN with VID == 2667 to IF -:eth1:-
Apr 04 14:15:23 reflecting-attraction ifup[5936]: /bin/ip addr add 10.245.184.20/255.255.255.0 broadcast 10.245.184.255 dev eth1.2667 label eth1.2667
Apr 04 14:15:23 reflecting-attraction ifup[5936]: /bin/ip link set dev eth1.2667 mtu 9000 up
Apr 04 14:15:23 reflecting-attraction ifup[5936]: RTNETLINK answers: Numerical result out of range

We can probably work around this by ensuring that the mtu is set correctly on the base interface as well.

[Andrew McDermott (frobware)]

The Juju bridge script, when creating the bridge device, generally leaves all iface options as it found them.

For example:

auto eth1.16
iface eth1.16 inet static
    address 10.245.184.100/24
    vlan-raw-device eth1
    mtu 1500
    vlan_id 16

becomes the following:

iface eth1.16 inet manual
    address 10.245.184.100/24
    vlan-raw-device eth1
    mtu 1500
    vlan_id 16

auto br-eth1.16
iface br-eth1.16 inet static
    address 10.245.184.100/24
    mtu 1500
    bridge_ports eth1.16
    bridge_stp off
    bridge_maxwait 0

Note: It doesn't propagate any bond options.

For example:

auto eth0
iface eth0 inet manual
    bond-lacp_rate slow
    bond-xmit_hash_policy layer2
    bond-miimon 100
    bond-master bond0
    mtu 1500
    bond-mode active-backup

[ skipping eth1 for clarity ]

auto bond0
iface bond0 inet dhcp
    bond-lacp_rate slow
    bond-xmit_hash_policy layer2
    bond-miimon 100
    mtu 1500
    bond-mode active-backup
    hwaddress 52:54:00:1c:f1:5b
    bond-slaves none

becomes the following when bridged:

auto eth0
iface eth0 inet manual
    bond-lacp_rate slow
    bond-xmit_hash_policy layer2
    bond-miimon 100
    bond-master bond0
    mtu 1500
    bond-mode active-backup

auto bond0
iface bond0 inet manual
    bond-lacp_rate slow
    bond-xmit_hash_policy layer2
    bond-miimon 100
    mtu 1500
    bond-mode active-backup
    hwaddress 52:54:00:1c:f1:5b
    bond-slaves none
    dns-nameservers 10.17.20.200
    dns-search maas19

auto br-bond0
iface br-bond0 inet dhcp
    mtu 1500
    hwaddress 52:54:00:1c:f1:5b
    bridge_ports bond0
    bridge_stp off
    bridge_maxwait 0
    dns-nameservers 10.17.20.200
    dns-search maas19

I generally propagated the existing options when creating the bridge device. Should MTU be an exception now?

[Ryan Harper (raharper)]

If you remove the eth1 manual stanza above, does it then work?

auto eth1
iface eth1 inet manual
    mtu 1500

[Mike Pontillo (mpontillo)]

In my testing on Xenial, you cannot set a child interface MTU to a value greater than its parent. For example, if eth0's MTU is 1500, setting eth0.100's MTU to 9000 (or 1501) causes:

SIOCSIFMTU: Numerical result out of range

This restriction makes sense when you consider the physical topology, since packets from eth0.100 would be sent via eth0. If the system attempts to send a packet larger than 1500 bytes via eth0 in this case, it would be dropped.

This issue makes me wish it were possible to set a "L3 MTU" (i.e., one that would be the default PMTU MSS, and fragment threshold for other packet types such as UDP and ICMP).

[Cassondra Foesch (puellanivis)]

I'm running into this issue as well. By moving my "mtu 9000" to the same "post-up ifconfig $IFACE mtu 9000" that I was using for all of my other interfaces, it at least started to actually bring the interface up, and set the gateway correctly. But it is still not setting the mtu properly for the vlan4 device, and only the vlan4 device.

(Irrelevant data trimmed from pastes below)

$ cat /etc/network/interfaces
# The primary network interface
auto enp3s0f0
iface enp3s0f0 inet manual
        bond-master bond0
        post-up ifconfig $IFACE mtu 9000

auto enp3s0f1
iface enp3s0f1 inet manual
        bond-master bond0
        post-up ifconfig $IFACE mtu 9000

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0f0 enp3s0f1
        post-up ifconfig $IFACE mtu 9000

auto mesh0
iface mesh0 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        post-up ifconfig $IFACE mtu 9000

auto vlan4
iface vlan4 inet static
        address 10.42.0.6
        netmask 255.255.255.0
        gateway 10.42.0.1
        post-up ifconfig $IFACE mtu 9000
        vlan-raw-device mesh0

$ systemctl status networking
● networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           └─50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Do 2016-05-19 07:22:30 UTC; 9min ago
     Docs: man:interfaces(5)
  Process: 3610 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 3341 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 3610 (code=exited, status=1/FAILURE)

Mai 19 07:22:27 gloss.orion.snow.mesh ifup[3610]: /sbin/ifup: waiting for lock on /run/network/ifstate.enp3s0f0
Mai 19 07:22:29 gloss.orion.snow.mesh ifup[3610]: /sbin/ifup: waiting for lock on /run/network/ifstate.enp3s0f1
Mai 19 07:22:29 gloss.orion.snow.mesh ifup[3610]: Waiting for mesh0 to get ready (MAXWAIT is 2 seconds).
Mai 19 07:22:30 gloss.orion.snow.mesh ifup[3610]: Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Mai 19 07:22:30 gloss.orion.snow.mesh ifup[3610]: RTNETLINK answers: File exists
Mai 19 07:22:30 gloss.orion.snow.mesh ifup[3610]: Failed to bring up vlan4.
Mai 19 07:22:30 gloss.orion.snow.mesh systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Mai 19 07:22:30 gloss.orion.snow.mesh systemd[1]: Failed to start Raise network interfaces.
Mai 19 07:22:30 gloss.orion.snow.mesh systemd[1]: networking.service: Unit entered failed state.
Mai 19 07:22:30 gloss.orion.snow.mesh systemd[1]: networking.service: Failed with result 'exit-code'.

$ /sbin/ifconfig
bond0 Link encap:Ethernet HWaddr 00:25:90:14:cb:d8
          inet6 addr: fe80::225:90ff:fe14:cbd8/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1
          RX packets:27794 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78498 errors:0 dropped:4 overruns:0 carrier:0
          collision...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kees Hoekzema (kees-r)]

I had/have sort of the same problem, when running the network startup it somehow wants to set up my vlans twice; i think when i do the 'master' interface, and once when it sets up the normal interface.

The first time it just sets up the vlan and assigns an address. The second time it tries to add that address again, but that obviously fails with an 'RTNETLINK answers: File exists'

And because the default (since 2014?) is to treat any error as fatal it just stops configuring the interface even tho it would be perfectly ok to at least try to set, for example, the default route or the 'up' statements. But no, a small step failed so it's better to leave the interface half configured.

On Ubuntu 16.04 this can be worked around by adding "EXCLUDE_INTERFACES=' --ignore-errors' " to /etc/default/networking so ifupdown does the sensible thing and tries to continue after an error instead of leaving a half configured interface.

[Dan Streetman (ddstreet)]

This is a side effect / duplicate of bug 1224007. The initial problem is the high vlan mtu setting, and the "File Exists" error is because ifup doesn't unwind on failure, so the interface is ip but not configured, which causes ifup to keep failing because it isn't expecting the interface to already be up.