JUJU Beta 10 VLAN config in ENI not adding 'auto iface'

[Ben]

Using MAAS 2Beta7 and JUJU 2 Beta 10

When deploying bare metal machine the interfaces file is missing a few 'auto iface' lines causing VLAN to not work after deploy. If you manually add the 'auto iface' to the interfaces file everything works correctly. This issue started with Beta 10, everything has been fine in previous Beta releases. Tested on multiple types of bare metal, this issue happens to them all.

##################################################
interfaces file after initial deploy.

auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

iface enp3s0f0 inet manual

auto br-enp3s0f0
iface br-enp3s0f0 inet static
    address 10.1.0.7/16
    gateway 10.1.0.1
    mtu 1500
    bridge_ports enp3s0f0

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

iface ens2f0.10 inet manual
    address 10.10.0.7/16
    vlan_id 10
    vlan-raw-device ens2f0
    mtu 1500

auto br-ens2f0.10
iface br-ens2f0.10 inet static
    address 10.10.0.7/16
    mtu 1500
    bridge_ports ens2f0.10

source /etc/network/interfaces.d/*.cfg
##################################################

VLAN works after you manually make the following changes

auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

auto enp3s0f0
iface enp3s0f0 inet manual

auto br-enp3s0f0
iface br-enp3s0f0 inet static
    address 10.1.0.7/16
    gateway 10.1.0.1
    mtu 1500
    bridge_ports enp3s0f0

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

auto ens2f0.10
iface ens2f0.10 inet manual
    address 10.10.0.7/16
    vlan_id 10
    vlan-raw-device ens2f0
    mtu 1500

auto br-ens2f0.10
iface br-ens2f0.10 inet static
    address 10.10.0.7/16
    mtu 1500
    bridge_ports ens2f0.10

source /etc/network/interfaces.d/*.cfg
##################################################

[Andrew McDermott (frobware)]

Please could you attach:

  /etc/network/interfaces-before-add-juju-bridge

[Andrew McDermott (frobware)]

Please could you run the following on your MAAS host:

$ apt-cache madison curtin

On my host I see the following output and I want to verify which version of curtin you have installed:

$ apt-cache madison curtin

    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://ppa.launchpad.net/maas/next/ubuntu xenial/main amd64 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://ppa.launchpad.net/maas/next/ubuntu xenial/main i386 Packages
    curtin | 0.1.0~bzr365-0ubuntu1 | http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
    curtin | 0.1.0~bzr365-0ubuntu1 | http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

[Andrew McDermott (frobware)]

In my setup:

 MAAS Version 2.0.0 (beta7+bzr5112)

 $ juju version
 2.0-beta10-xenial-amd64

I am unable to reproduce this:

My /etc/network/interfaces has 1 VLAN:

buntu@prealtar-erinn:~$ cat /etc/network/interfaces
auto lo
iface lo inet loopback
    dns-nameservers 10.12.20.2
    dns-search maas20

iface eth0 inet manual

auto br-eth0
iface br-eth0 inet static
    address 10.12.20.3/24
    gateway 10.12.20.1
    mtu 1500
    bridge_ports eth0
    dns-nameservers 10.12.20.2

iface eth0.110 inet manual
    address 192.168.110.2/24
    vlan-raw-device eth0
    mtu 1500
    vlan_id 110

auto br-eth0.110
iface br-eth0.110 inet static
    address 192.168.110.2/24
    mtu 1500
    bridge_ports eth0.110

source /etc/network/interfaces.d/*.cfg

and I see that the VLAN is up (192.168.110.2):

ubuntu@prealtar-erinn:~$ ifconfig -a
br-eth0 Link encap:Ethernet HWaddr 52:54:00:46:62:3c
          inet addr:10.12.20.3 Bcast:10.12.20.255 Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe46:623c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:36100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30142 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:77383443 (77.3 MB) TX bytes:2361754 (2.3 MB)

br-eth0.110 Link encap:Ethernet HWaddr 52:54:00:46:62:3c
          inet addr:192.168.110.2 Bcast:192.168.110.255 Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe46:623c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

eth0 Link encap:Ethernet HWaddr 52:54:00:46:62:3c
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:48881 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31432 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:84076902 (84.0 MB) TX bytes:2520826 (2.5 MB)

eth0.110 Link encap:Ethernet HWaddr 52:54:00:46:62:3c
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:34320 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34320 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:36057759 (36.0 MB) TX bytes:36057759 (36.0 MB)

[Cheryl Jennings (cherylj)]

Marking as incomplete until we get /etc/network/interfaces-before-add-juju-bridge as requested in comment #1.

[Ben (bjenkins-x)]

Ok, I did a fresh install on completely different hardware just to make sure that The issue still persists. I did a simple single interface with an untagged network and a single VLAN. The VLAN still does not work until I go to ENI and add the 'auto iface' to both places where it is missing.

Here is the requested data.

####################################################

here is the apt-cache madison curtin

    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://ppa.launchpad.net/maas/next/ubuntu xenial/main amd64 Packages
    curtin | 0.1.0~bzr389-0ubuntu1~16.04.1 | http://ppa.launchpad.net/maas/next/ubuntu xenial/main i386 Packages
    curtin | 0.1.0~bzr365-0ubuntu1 | http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
    curtin | 0.1.0~bzr365-0ubuntu1 | http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

####################################################

Here is the interfaces file after deploy

auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

iface enp3s0f0 inet manual

auto br-enp3s0f0
iface br-enp3s0f0 inet static
    address 10.1.0.4/16
    gateway 10.1.0.1
    mtu 1500
    bridge_ports enp3s0f0

auto enp3s0f1
iface enp3s0f1 inet manual
    mtu 1500

auto enp4s0f0
iface enp4s0f0 inet manual
    mtu 1500

auto enp4s0f1
iface enp4s0f1 inet manual
    mtu 1500

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

auto ens2f1
iface ens2f1 inet manual
    mtu 1500

iface enp3s0f0.10 inet manual
    address 10.10.0.4/16
    mtu 1500
    vlan_id 10
    vlan-raw-device enp3s0f0

auto br-enp3s0f0.10
iface br-enp3s0f0.10 inet static
    address 10.10.0.4/16
    mtu 1500
    bridge_ports enp3s0f0.10

source /etc/network/interfaces.d/*.cfg

####################################################

[Andrew McDermott (frobware)]

I looked at the difference between the output from beta9 and beta10 and I see the following:

$ diff -u /tmp/ENI/9.eni /tmp/ENI/10.eni
--- /tmp/ENI/9.eni 2016-07-07 13:42:46.910826658 +0100
+++ /tmp/ENI/10.eni 2016-07-07 13:42:46.938826920 +0100
@@ -3,6 +3,8 @@
     dns-nameservers 10.1.0.2
     dns-search maas

+iface enp3s0f0 inet manual
+
 auto br-enp3s0f0
 iface br-enp3s0f0 inet static
     address 10.1.0.4/16
@@ -30,9 +32,14 @@
 iface ens2f1 inet manual
     mtu 1500

+iface enp3s0f0.10 inet manual
+ address 10.10.0.4/16
+ mtu 1500
+ vlan_id 10
+ vlan-raw-device enp3s0f0
+
 auto br-enp3s0f0.10
 iface br-enp3s0f0.10 inet static
     address 10.10.0.4/16
     mtu 1500
- vlan-raw-device enp3s0f0
     bridge_ports enp3s0f0.10

***** I don't see any removal or addition of 'auto' stanzas.

On top of that here are the differences against some previous beta versions:

$ diff /tmp/ENI/6.eni /tmp/ENI/10.eni
14,15d13
< bridge_stp off
< bridge_maxwait 0
48,49d45
< bridge_stp off
< bridge_maxwait 0

$ diff /tmp/ENI/7.eni /tmp/ENI/10.eni

$ diff /tmp/ENI/8.eni /tmp/ENI/10.eni

[Andrew McDermott (frobware)]

Given that there are no real differences regarding missing 'auto' stanzas since beta6 I am marking this as invalid.

[Ben (bjenkins-x)]

Please don't stop looking into this issue. It is a real show stopper for a MAAS/Juju environment. I have tried on LAB and production equipment, the issue happens on both. MAAS version beta 7 through RC1 and JUJU Beta 11. Every physical machine that is deployed with VLANS must be manually altered to work correctly and stops the ability to quickly deploy and scale. Let me know what data I can gather for you to help find out why this is happening. I did not have the problem at all with beta 9 and earlier versions. I have been deploying only Xenial physical machines if that helps.

[Andrew McDermott (frobware)]

If this is still happening then I can I ask that you always include all of:

  /etc/network/interfaces* (directories too)

so that we can see the before and after.

One thing that I would like to check is: if you're using a VLAN is the parent device always configured in MAAS to have an address? If not you will run into:

  https://bugs.launchpad.net/juju-core/+bug/1566791

[Andrew McDermott (frobware)]

Given my observations in comment #6 there have been no removals/additions for 'auto' stanzas when creating the bridges for many beta releases now. Would it be possible for you to go back to beta6 and see if it happens there too?

[Andrew McDermott (frobware)]

Also feel free to reach out to me (frobware) on freenode in channel #juju.

[Andrew McDermott (frobware)]

Please could you include dpkg info for the ifupdown package.

> On 7 Jul 2016, at 16:07, Ben <email address hidden> wrote:
>
> Please don't stop looking into this issue. It is a real show stopper
> for a MAAS/Juju environment. I have tried on LAB and production
> equipment, the issue happens on both. MAAS version beta 7 through RC1
> and JUJU Beta 11. Every physical machine that is deployed with VLANS
> must be manually altered to work correctly and stops the ability to
> quickly deploy and scale. Let me know what data I can gather for you to
> help find out why this is happening. I did not have the problem at all
> with beta 9 and earlier versions. I have been deploying only Xenial
> physical machines if that helps.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1596597
>
> Title:
> JUJU Beta 10 VLAN config in ENI not adding 'auto iface'
>
> Status in juju-core:
> Invalid
>
> Bug description:
> Using MAAS 2Beta7 and JUJU 2 Beta 10
>
> When deploying bare metal machine the interfaces file is missing a few
> 'auto iface' lines causing VLAN to not work after deploy. If you
> manually add the 'auto iface' to the interfaces file everything works
> correctly. This issue started with Beta 10, everything has been fine
> in previous Beta releases. Tested on multiple types of bare metal,
> this issue happens to them all.
>
> ##################################################
> interfaces file after initial deploy.
>
> auto lo
> iface lo inet loopback
> dns-nameservers 10.1.0.2
> dns-search maas
>
> iface enp3s0f0 inet manual
>
> auto br-enp3s0f0
> iface br-enp3s0f0 inet static
> address 10.1.0.7/16
> gateway 10.1.0.1
> mtu 1500
> bridge_ports enp3s0f0
>
> auto ens2f0
> iface ens2f0 inet manual
> mtu 1500
>
> iface ens2f0.10 inet manual
> address 10.10.0.7/16
> vlan_id 10
> vlan-raw-device ens2f0
> mtu 1500
>
> auto br-ens2f0.10
> iface br-ens2f0.10 inet static
> address 10.10.0.7/16
> mtu 1500
> bridge_ports ens2f0.10
>
> source /etc/network/interfaces.d/*.cfg
> ##################################################
>
> VLAN works after you manually make the following changes
>
> auto lo
> iface lo inet loopback
> dns-nameservers 10.1.0.2
> dns-search maas
>
> auto enp3s0f0
> iface enp3s0f0 inet manual
>
> auto br-enp3s0f0
> iface br-enp3s0f0 inet static
> address 10.1.0.7/16
> gateway 10.1.0.1
> mtu 1500
> bridge_ports enp3s0f0
>
> auto ens2f0
> iface ens2f0 inet manual
> mtu 1500
>
> auto ens2f0.10
> iface ens2f0.10 inet manual
> address 10.10.0.7/16
> vlan_id 10
> vlan-raw-device ens2f0
> mtu 1500
>
> auto br-ens2f0.10
> iface br-ens2f0.10 inet static
> address 10.10.0.7/16
> mtu 1500
> bridge_ports ens2f0.10
>
> source /etc/network/interfaces.d/*.cfg
> ##################################################
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1596597/+subscriptions

[Andrew McDermott (frobware)]

WIP - https://github.com/frobware/juju/tree/master-1596597

[Andrew McDermott (frobware)]

@bjenkins-x - there is a work-in-progress binary you could try here:

 http://178.62.20.154/~aim/juju-2.0-beta11+lp1596597.tar.xz

it's not clear to me why this change is needed so it would be great if you could confirm whether this change will work for your setup.

[Ben (bjenkins-x)]

Will test shortly and report back. Thank you so much for keeping this issue open.

[Andrew McDermott (frobware)]

RB - http://reviews.vapour.ws/r/5235/

[Ben (bjenkins-x)]

Sorry, the fix http://178.62.20.154/~aim/juju-2.0-beta11+lp1596597.tar.xz did not help with the interface issue.

I simplified my testing environments as much as possible to test the new binaries listed above.

There are now 3 physical servers.
1) MAAS (RC2) and JUJU 2 beta11+lp1596597
2) JUJU bootstrap server
3) simple production machine deployed using juju add-machine

The auto iface is still not added after deploy to the parent device for the physical NIC or the VLAN device. The BR devices do have auto but the parents do not. The VLAN is on an interface with a static IP.

I deployed a fresh server, updated everything, installed MAAS2RC2, installed JUJU2 then overwrote the files in /usr/lib/juju-2.0/bin/ with the files from http://178.62.20.154/~aim/juju-2.0-beta11+lp1596597.tar.xz.

If I did something wrong in testing please let me know but VLAN will not work until I manually add auto enp3s0f0.10 to the parent interface.

Here is the ENI file after deploy

###############################################

auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

iface enp3s0f0 inet manual

auto br-enp3s0f0
iface br-enp3s0f0 inet static
    gateway 10.1.0.1
    address 10.1.0.4/16
    mtu 1500
    bridge_ports enp3s0f0

auto enp3s0f1
iface enp3s0f1 inet manual
    mtu 1500

auto enp4s0f0
iface enp4s0f0 inet manual
    mtu 1500

auto enp4s0f1
iface enp4s0f1 inet manual
    mtu 1500

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

auto ens2f1
iface ens2f1 inet manual
    mtu 1500

iface enp3s0f0.10 inet manual
    address 10.10.0.4/16
    vlan_id 10
    mtu 1500
    vlan-raw-device enp3s0f0

auto br-enp3s0f0.10
iface br-enp3s0f0.10 inet static
    address 10.10.0.4/16
    mtu 1500
    bridge_ports enp3s0f0.10

source /etc/network/interfaces.d/*.cfg

###############################################

Here is the interfaces-before-add-juju-bridge

###############################################

auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

auto enp3s0f0
iface enp3s0f0 inet static
    gateway 10.1.0.1
    address 10.1.0.4/16
    mtu 1500

auto enp3s0f1
iface enp3s0f1 inet manual
    mtu 1500

auto enp4s0f0
iface enp4s0f0 inet manual
    mtu 1500

auto enp4s0f1
iface enp4s0f1 inet manual
    mtu 1500

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

auto ens2f1
iface ens2f1 inet manual
    mtu 1500

auto enp3s0f0.10
iface enp3s0f0.10 inet static
    address 10.10.0.4/16
    vlan_id 10
    mtu 1500
    vlan-raw-device enp3s0f0

source /etc/network/interfaces.d/*.cfg

###############################################

[Andrew McDermott (frobware)]

I reworked this a little today.

New checksum for the binary is:

-rw-rw-r-- 1 aim aim 51186328 Jul 13 21:28 juju-2.0-beta11+lp1596597.tar.xz

$ md5sum juju-2.0-beta11+lp1596597.tar.xz
81c08fb58b92718f55f8800850161ca8 juju-2.0-beta11+lp1596597.tar.xz

[Andrew McDermott (frobware)]

Here's the new transformation based on the original file in comment #17

aim@spicy:~/juju (master-lp1596597)
$ ./provider/maas/add-juju-bridge.py ~/x.eni
auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.2
    dns-search maas

auto enp3s0f0
iface enp3s0f0 inet manual

auto br-enp3s0f0
iface br-enp3s0f0 inet static
    gateway 10.1.0.1
    address 10.1.0.4/16
    mtu 1500
    bridge_ports enp3s0f0

auto enp3s0f1
iface enp3s0f1 inet manual
    mtu 1500

auto enp4s0f0
iface enp4s0f0 inet manual
    mtu 1500

auto enp4s0f1
iface enp4s0f1 inet manual
    mtu 1500

auto ens2f0
iface ens2f0 inet manual
    mtu 1500

auto ens2f1
iface ens2f1 inet manual
    mtu 1500

auto enp3s0f0.10
iface enp3s0f0.10 inet manual
    address 10.10.0.4/16
    vlan_id 10
    mtu 1500
    vlan-raw-device enp3s0f0

auto br-enp3s0f0.10
iface br-enp3s0f0.10 inet static
    address 10.10.0.4/16
    mtu 1500
    bridge_ports enp3s0f0.10

[Ben (bjenkins-x)]

I pulled the latest version from github and everything works great now. juju status shows model version beta12.1.

Thank you so much for not giving up on this!

[Michael Foord (mfoord)]

Fixed in 1.25 as part of: https://github.com/juju/juju/pull/6013