[2.2.1] When deploying servers with VLAN using IPv6 alias ifdown fails with cannot find device

[Ben]

Using MAAS 2.2.1 a server configured with an alias to add IPv6 will not allow ifdown to function correctly. You will receive cannot find device. This is a problem when paired with JUJU as you can no longer create bridges for charms because the last step before activating the new interfaces.new file is to bring the interfaces down but the error stops the process. This was not a problem on 2.1.x or 2.0.x.

To recreate, commission a server and add an IPv4 VLAN subnet to the root interface. Then add an alias to the VLAN interface using IPv6 subnet. Deploy the server using Ubuntu 16.04 and try to ifdown the VLAN interface.

root@server01:/etc/network# ifconfig
enp2s0f0 Link encap:Ethernet HWaddr fc:aa:14:1d:xx:xx
          inet addr:10.1.0.7 Bcast:10.1.255.255 Mask:255.255.0.0
          inet6 addr: fe80::feaa:14ff:fe1d:xxxx/64 Scope:Link
          inet6 addr: xxxx:xxxx:xxxx:2:0:1:0:2/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:5454 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1394 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6681566 (6.6 MB) TX bytes:195223 (195.2 KB)

enp2s0f0.11 Link encap:Ethernet HWaddr fc:aa:14:1d:xx:xx
          inet addr:10.11.0.4 Bcast:10.11.255.255 Mask:255.255.0.0
          inet6 addr: xxxx:xxxx:xxxx:3:0:1:0:2/64 Scope:Global
          inet6 addr: fe80::feaa:14ff:fe1d:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:24 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4820 (4.8 KB) TX bytes:756 (756.0 B)

root@server01:/etc/network# ifdown enp2s0f0.11
Removed VLAN -:enp2s0f0.11:-
Cannot find device "enp2s0f0.11"

root@server01:/etc/network# ifup enp2s0f0.11
/etc/network/if-pre-up.d/mtuipv6: line 9: /sys/class/net/enp2s0f0.11/mtu: No such file or directory
/etc/network/if-pre-up.d/mtuipv6: line 10: /proc/sys/net/ipv6/conf/enp2s0f0.11/mtu: No such file or directory
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
ifup: recursion detected for interface enp2s0f0 in parent-lock phase
Added VLAN with VID == 11 to IF -:enp2s0f0:-
Waiting for DAD... Done

[Andres Rodriguez (andreserl)]

Hi Ben,

Could you please attach the output of:

 - maas <user> machines get-curtin-config <system_id>

This will tell us the network configuration that was supposed to be made. That said, this could be an issue with either curtin (which actually wrote e/n/i) or ifupdown (the latest in xenial).

[Ben (bjenkins-x)]

Here you go, sure hope this helps. Let me know what else I can do to help track this issue down. If I do not use IPv6 there are no issues with the ifdown or deploying juju charms.

early_commands:
  driver_00:
  - sh
  - -c
  - echo third party drivers not installed or necessary.
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}
  package: linux-lowlatency-hwe-16.04
late_commands:
  maas:
  - wget
  - --no-proxy
  - http://10.1.0.2:5240/MAAS/metadata/latest/by-id/fyfbct/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: enp2s0f0
    mac_address: fc:aa:14:1d:xx:xx
    mtu: 9000
    name: enp2s0f0
    subnets:
    - address: 10.1.0.7/16
      dns_nameservers:
      - 10.1.0.2
      gateway: 10.1.0.1
      type: static
    - address: xxxx:xxxx:xxxx:2:0:1:0:2/64
      dns_nameservers:
      - xxxx:xxxx:xxxx:2::2
      gateway: xxxx.xxxx.xxxx:2::1
      type: static
    type: physical
  - id: enp2s0f0.11
    mtu: 9000
    name: enp2s0f0.11
    subnets:
    - address: 10.11.0.4/16
      dns_nameservers: []
      type: static
    - address: xxxx.xxxx.xxxx:3:0:1:0:2/64
      dns_nameservers: []
      type: static
    type: vlan
    vlan_id: 11
    vlan_link: enp2s0f0
  - id: enp2s0f0.21
    mtu: 9000
    name: enp2s0f0.21
    subnets:
    - address: 10.21.0.4/16
      dns_nameservers: []
      type: static
    - address: xxxx.xxxx.xxxx:4:0:1:0:2/64
      dns_nameservers: []
      type: static
    type: vlan
    vlan_id: 21
    vlan_link: enp2s0f0
  - id: enp2s0f0.31
    mtu: 9000
    name: enp2s0f0.31
    subnets:
    - address: 10.31.0.3/16
      dns_nameservers: []
      type: static
    - address: xxxx.xxxx.xxxx:5:0:1:0:1/64
      dns_nameservers: []
      type: static
    type: vlan
    vlan_id: 31
    vlan_link: enp2s0f0
  - id: enp2s0f1
    mac_address: fc:aa:14:1d:db:c9
    mtu: 9000
    name: enp2s0f1
    subnets:
    - type: manual
    type: physical
  - id: ens1
    mac_address: e4:1d:2d:21:10:50
    mtu: 9000
    name: ens1
    subnets:
    - type: manual
    type: physical
  - id: ens1.3
    mtu: 9000
    name: ens1.3
    subnets:
    - address: 10.3.0.3/16
      dns_nameservers: []
      type: static
    - address: xxxx.xxxx.xxxx:7:0:1:0:1/64
      dns_nameservers: []
      type: static
    type: vlan
    vlan_id: 3
    vlan_link: ens1
  - id: ens1.4
    mtu: 9000
    name: ens1.4
    subnets:
    - address: 10.4.0.3/16
      dns_nameservers: []
      type: static
    - address: xxxx.xxxx.xxxx:8:0:1:0:1/64
      dns_nameservers: []
      type: static
    type: vlan
    vlan_id: 4
    vlan_link: ens1
  - address:
    - 10.1.0.2
    search:
    - deploy.vorbi.com
    - maas
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  - custom
partitioning_commands:
  builtin:
  - curtin
  - block-meta
  - custom
reporting:
  maas:
    consumer_key: xxxx
    endpoint: http://10.1.0.2:5240/MAAS/metadata/status/fyfbct
    token_key: xxxx
    token_secret: xxxx
    type: webhook
storage:
  config:
  - grub_device: true
    id: sda
    model: Samsung SSD 850
    name: sda
    ptable: msdos
    serial: S24ZNWAG932461X
    type: disk
    wi...

[Ryan Harper (raharper)]

Hello,

The ifupdown scripts are pretty noisy, so there's a log of debug level messages coming through.

For example,

ifdown enp2s0f0.11

Will
 - invoke `ip addr del` to remove the v4 address
 - the vlan down script will remove the vlan on device enp2s0f0.11
 - invoke `ip -6 addr del` to remove the v6 address

When the v6 addr del runs, the vlan script has already removed the device, so it shows the error message that you observed, but it's non fatal.

The return code after ifdown enp2s0f0.11 (echo $?) should show zero, and ifconfig should confirm that enp2s0f0.11 interface is no longer present.

The following ifup is also noisy but not fatal. Can you confirm the return code from ifup is zero and that ifconfig output shows the interface properly configured?

I can see similar output but the end result when I'm testing has the interfaces configured properly.

[Ben (bjenkins-x)]

The interfaces do function correctly after manual intervention even with the errors but my problem, from what I understand about MAAS is the following

root@server01:/etc/network# ifdown enp2s0f0.11
  Removed VLAN -:enp2s0f0.11:-
  Cannot find device "enp2s0f0.11"

is enough to stop the bridges from being constructed correctly. I hope someone from the MAAS team can step in here with a little more detail on why it fails completely because of this error. This has not been an issue with previous versions of MAAS. I get the interfaces.new file with all the correct settings but it is never copied to interfaces and the network is not brought down and back up to load the new configuration. I have to manually do the interface.new to interfaces copy and reboot to get the interfaces to function correctly.

[Ryan Harper (raharper)]

On Mon, Jul 17, 2017 at 2:51 PM, Ben <email address hidden> wrote:

> The interfaces do function correctly after manual intervention even with
> the errors but my problem, from what I understand about MAAS is the
> following
>
> root@server01:/etc/network# ifdown enp2s0f0.11
> Removed VLAN -:enp2s0f0.11:-
> Cannot find device "enp2s0f0.11"
>

> is enough to stop the bridges from being constructed correctly. I hope
>

This sounds fatal; can you confirm the return code?

ifdown enp2s0f0.11; echo $?

> someone from the MAAS team can step in here with a little more detail on
> why it fails completely because of this error. This has not been an
> issue with previous versions of MAAS. I get the interfaces.new file
> with all the correct settings but it is never copied to interfaces and
> the network is not brought down and back up to load the new
> configuration. I have to manually do the interface.new to interfaces
> copy and reboot to get the interfaces to function correctly.
>

The reconfiguration of the network files is done by JUJU; let's see if they
can shed some light here on how they're doing their network configuration
modifications.

When the vlan ifdown script (/etc/network/if-down.d/vlan) is called a
second time (one for each iface stanza, there is a v4 and a v6 stanza in
/etc/network/interfaces
for the two IP configurations on the vlan in question) the vlan device is
already deleted. This is expected and in my testing, it shows to be
non-fatal but noisy.

# ifdown ens4.11; echo $?
Removed VLAN -:ens4.11:-
Cannot find device "ens4.11"
ERROR: trying to remove VLAN -:ens4.11:- error: No such device
run-parts: /etc/network/if-post-down.d/vlan exited with return code 4
0

The vlan hook exits non-zero, but ifdown returns 0. And functionally
the vlan interface is no long present.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1703689
>
> Title:
> [2.2.1] When deploying servers with VLAN using IPv6 alias ifdown fails
> with cannot find device
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1703689/+subscriptions
>

[Ben (bjenkins-x)]

ubuntu@server01:~$ sudo ifdown enp2s0f0.11; echo $?
Removed VLAN -:enp2s0f0.11:-
Cannot find device "enp2s0f0.11"
0

ubuntu@server01:~$ sudo ifup enp2s0f0.11; echo $?
/etc/network/if-pre-up.d/mtuipv6: line 9: /sys/class/net/enp2s0f0.11/mtu: No such file or directory
/etc/network/if-pre-up.d/mtuipv6: line 10: /proc/sys/net/ipv6/conf/enp2s0f0.11/mtu: No such file or directory
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
ifup: recursion detected for interface enp2s0f0 in parent-lock phase
Added VLAN with VID == 11 to IF -:enp2s0f0:-
Waiting for DAD... Done
0

Thanks for helping move this forward!

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]