Xenial Deploy fails when using /etc/network/interface

[john]

MAAS 2.2.2 Move from Curtin 505 to 532 breaks use of /etc/network/interface.

Our OVS OpenStack charm modifies /etc/network/interfaces to set up OVS and adds ethtool tso=off,tro=off, etc.
We've deployed this charm in production at PCCW. PCCW updated their MAAS server and the charm no longer works.

in curtin 532, /etc/network//interfaces is empty. The new definitions are in /etc/network/interfaces.d/50-cloud-init.cfg

Attaching charm code and juju-log file

[john (g-john-p)]

juju deploy log file: http://paste.ubuntu.com/25961066/

our charm code https://github.com/cplane-networks/dvnd-juju/tree/master/neutron-openvswitch-cplane

[Andres Rodriguez (andreserl)]

Curtin has changed the way how it does networking and it now tells cloud-init to do the networking. So whatever you used in late_commands is no longer going to work for configuring interfaces.

[Ryan Harper (raharper)]

Can you attach a working eni file? I'd like to capture what additional elements in eni you're using that aren't modeled in the network configuration in MAAS.

[john (g-john-p)]

I don't know what an eni file is. can you please give more instruction.
We are not using any late_commands. Everything is done using juju charms. please see our charms code.

[Andres Rodriguez (andreserl)]

@John,

Please provide the configuration steps you are doing to modify e/n/i, as well as provide the resulting one on a deployed machine.

That said, the quickest work around would be to revert curtin to the older version, which would allow you to continue to use your current configuration.

[Ryan Harper (raharper)]

Sorry,

If you can provide a copy of /etc/network/interfaces
/etc/network/interfaces.d/* files one of the deployed nodes where
it is not broken.

Ryan

On Tue, Nov 14, 2017 at 11:26 AM, john <email address hidden> wrote:

> I don't know what an eni file is. can you please give more instruction.
> We are not using any late_commands. Everything is done using juju charms.
> please see our charms code.
>
> ** Changed in: curtin
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1732202
>
> Title:
> Xenial Deploy fails when using /etc/network/interface
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1732202/+subscriptions
>

[Ryan Harper (raharper)]

2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/install",
line 240, in <module>
2017-11-14 04:40:55 INFO install main()
2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/install",
line 233, in main
2017-11-14 04:40:55 INFO install hooks.execute(sys.argv)
2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/charmhelpers/core/hookenv.py",
line 715, in execute
2017-11-14 04:40:55 INFO install self._hooks[hook_name]()
2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/install",
line 222, in install
2017-11-14 04:40:55 INFO install gw=config('data-gateway'))
2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/cplane_network.py",
line 68, in add_bridge
2017-11-14 04:40:55 INFO install read_only=False)
2017-11-14 04:40:55 INFO install File
"/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/cplane_interface.py",
line 93, in extract_net_config
2017-11-14 04:40:55 INFO install format(name=ifname))
2017-11-14 04:40:55 INFO install ValueError: eth1.10: Could not parse
needed iface params

Your charm only reads /etc/network/interfaces

However, to properly read all network config you need to combine:

/etc/network/interfaces and any *.cfg in /etc/network/interfaces.d

https://github.com/cplane-networks/dvnd-juju/blob/master/neutron-openvswitch-cplane/hooks/cplane_interface.py#L36
'

That code only works for eni (/etc/network/interface) files which don't
also source additional configuration.

On Tue, Nov 14, 2017 at 11:39 AM, Ryan Harper <email address hidden>
wrote:

> Sorry,
>
> If you can provide a copy of /etc/network/interfaces
> /etc/network/interfaces.d/* files one of the deployed nodes where
> it is not broken.
>
> Ryan
>
>
> On Tue, Nov 14, 2017 at 11:26 AM, john <email address hidden> wrote:
>
>> I don't know what an eni file is. can you please give more instruction.
>> We are not using any late_commands. Everything is done using juju
>> charms. please see our charms code.
>>
>> ** Changed in: curtin
>> Status: Incomplete => New
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1732202
>>
>> Title:
>> Xenial Deploy fails when using /etc/network/interface
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/curtin/+bug/1732202/+subscriptions
>>
>
>

[john (g-john-p)]

attaching working /etc/network/interfaces from curtin 505

[john (g-john-p)]

I just added /etc/network/interfaces.

/etc/network/interfaces/* was empty.

John Casey
CPlane Networks
Direct: 415-215-0854

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ryan Harper
Sent: Tuesday, November 14, 2017 9:39 AM
To: John Casey <email address hidden>
Subject: Re: [Bug 1732202] Re: Xenial Deploy fails when using /etc/network/interface

Sorry,

If you can provide a copy of /etc/network/interfaces
/etc/network/interfaces.d/* files one of the deployed nodes where it is not broken.

Ryan

On Tue, Nov 14, 2017 at 11:26 AM, john <email address hidden> wrote:

> I don't know what an eni file is. can you please give more instruction.
> We are not using any late_commands. Everything is done using juju charms.
> please see our charms code.
>
> ** Changed in: curtin
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to the
> bug report.
> https://bugs.launchpad.net/bugs/1732202
>
> Title:
> Xenial Deploy fails when using /etc/network/interface
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1732202/+subscriptions
>

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1732202

Title:
  Xenial Deploy fails when using /etc/network/interface

Status in curtin:
  New
Status in MAAS:
  Invalid

Bug description:
  MAAS 2.2.2 Move from Curtin 505 to 532 breaks use of
  /etc/network/interface.

  Our OVS OpenStack charm modifies /etc/network/interfaces to set up OVS and adds ethtool tso=off,tro=off, etc.
  We've deployed this charm in production at PCCW. PCCW updated their MAAS server and the charm no longer works.

  in curtin 532, /etc/network//interfaces is empty. The new
  definitions are in /etc/network/interfaces.d/50-cloud-init.cfg

  Attaching charm code and juju-log file

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1732202/+subscriptions

[john (g-john-p)]

/etc/network/interfaces.d/ was empty.

[Ryan Harper (raharper)]

Thanks for the input file, do you have the original/backup file before it was modified?

/etc/network/interfaces.cplane.old

?

I'm interested in seeing the transformation that the charm is doing to the base configuration.

[Ryan Harper (raharper)]

> /etc/network/interfaces.d/ was empty.

On the older curtin, which wrote eni directly, the interfaces.d dir will be empty as you found.
With newer curtin, the network configuration is given to cloud-init to render and it will write out to /etc/network/interfaces.d/50-cloud-init.cfg

If the charm were to combine /etc/network/interface with any files in /etc/network/interfaces.d/*.cfg ; then you'd have the full eni configuration to operate upon like in the previous case.

I suspect, as a quick check, if you read in /etc/network/interfaces.d/50-cloud-init.cfg as a possible source of network config, the charm should just work;

[Ryan Harper (raharper)]

https://github.com/raharper/dvnd-juju/pull/1

On newer Ubuntu systems, the network configuration file may be generated by
cloud-init and written to /etc/network/interfaces.d/50-cloud-init.cfg

If present, use that, otherwise fallback to reading /etc/network/interfaces.

Note, long term, this code needs to read /etc/network/interfaces, and for each
sourcedir directive in eni, search those directories for additional configurations
to capture all possible network configuration that's under control of ifupdown.

[john (g-john-p)]

thanks Ryan. We will update our charms to incorporate this logic.

[john (g-john-p)]

Hi Ryan,

We are now running into another issue. it seems that interface that are placed in /etc/network/interfaces.d/50-cloud-init.cfg are not supported using the ifup and ifdown.

I believe ifup and ifdown look into eni. Is there a different set of commands to use when no using eni?

[john (g-john-p)]

Hi Ryan: here is more info. All of what I'm gong to share is different in curtin 532 vs 505. ( our charms are working perfectly in 505)

In our charm, we need to set the MTU and TSO, GSO, TRO, etc of each interface (see charm)

We’ve already made a fix to edit the /etc/network/interfaces.d/50-cloud-init.cfg file.

But now we need to ifdown/ifup the interfaces after we set the MTU etc.

We are now getting this error.

og Change request for mtu for interface eth1.10 = 9000
2017-11-16 03:50:57 INFO config-changed ifdown: interface eth1.10 not configured
2017-11-16 03:50:57 INFO config-changed
2017-11-16 03:50:57 INFO config-changed Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
2017-11-16 03:50:57 INFO config-changed RTNETLINK answers: Invalid argument
2017-11-16 03:50:57 INFO config-changed RTNETLINK answers: Numerical result out of range
2017-11-16 03:50:57 INFO config-changed Failed to bring up eth1.10.
2017-11-16 03:50:57 INFO config-changed Traceback (most recent call last):
2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 240, in <module>
2017-11-16 03:50:57 INFO config-changed main()
2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 233, in main
2017-11-16 03:50:57 INFO config-changed hooks.execute(sys.argv)
2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/charmhelpers/core/hookenv.py", line 715, in execute
2017-11-16 03:50:57 INFO config-changed self._hooks[hook_name]()
2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 137, in config_changed
2017-11-16 03:50:57 INFO config-changed change_iface_config(interface[0], 'mtu', interface[1])
2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-neutron-openvswitch-cplane-0/charm/hooks/cplane_network.py", line 127, in change_iface_config
2017-11-16 03:50:57 INFO config-changed subprocess.check_call(cmd)
2017-11-16 03:50:57 INFO config-changed File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
2017-11-16 03:50:57 INFO config-changed raise CalledProcessError(retcode, cmd)
2017-11-16 03:50:57 INFO config-changed subprocess.CalledProcessError: Command '['ifup', u'eth1.10']' returned non-zero exit st

When I go into the server and try to do an ifup/down manually I’m getting this error
eth1.10 no such interface

ubuntu@MACH4:~$ sudo -i
root@MACH4:~# ifdown eth1.10
ifdown: interface eth1.10 not configured

but it clearly shows it is

eth1.10 Link encap:Ethernet HWaddr 52:54:00:5a:57:aa
          inet6 addr: fe80::5054:ff:fe5a:57aa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:185 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12139 (12.1 KB) TX bytes:21723 (21.7 KB)

We also just noticed that the ovs interface ...

[john (g-john-p)]

Syslog http://paste.ubuntu.com/25972961/
machine2-log http://paste.ubuntu.com/25972963/
charm log http://paste.ubuntu.com/25972970/

[Ryan Harper (raharper)]

Can you attach a tar of /etc/network/ ?

That way I can see what ifupdown is consuming.

On Thu, Nov 16, 2017 at 2:04 AM, john <email address hidden> wrote:

> Hi Ryan: here is more info. All of what I'm gong to share is different
> in curtin 532 vs 505. ( our charms are working perfectly in 505)
>
>
> In our charm, we need to set the MTU and TSO, GSO, TRO, etc of each
> interface (see charm)
>
> We’ve already made a fix to edit the /etc/network/interfaces.d/50-cloud-
> init.cfg file.
>
> But now we need to ifdown/ifup the interfaces after we set the MTU etc.
>
> We are now getting this error.
>
> og Change request for mtu for interface eth1.10 = 9000
> 2017-11-16 03:50:57 INFO config-changed ifdown: interface eth1.10 not
> configured
> 2017-11-16 03:50:57 INFO config-changed
> 2017-11-16 03:50:57 INFO config-changed Set name-type for VLAN subsystem.
> Should be visible in /proc/net/vlan/config
> 2017-11-16 03:50:57 INFO config-changed RTNETLINK answers: Invalid argument
> 2017-11-16 03:50:57 INFO config-changed RTNETLINK answers: Numerical
> result out of range
> 2017-11-16 03:50:57 INFO config-changed Failed to bring up eth1.10.
> 2017-11-16 03:50:57 INFO config-changed Traceback (most recent call last):
> 2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-
> neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 240, in
> <module>
> 2017-11-16 03:50:57 INFO config-changed main()
> 2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-
> neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 233, in
> main
> 2017-11-16 03:50:57 INFO config-changed hooks.execute(sys.argv)
> 2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-
> neutron-openvswitch-cplane-0/charm/hooks/charmhelpers/core/hookenv.py",
> line 715, in execute
> 2017-11-16 03:50:57 INFO config-changed self._hooks[hook_name]()
> 2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-
> neutron-openvswitch-cplane-0/charm/hooks/config-changed", line 137, in
> config_changed
> 2017-11-16 03:50:57 INFO config-changed change_iface_config(interface[0],
> 'mtu', interface[1])
> 2017-11-16 03:50:57 INFO config-changed File "/var/lib/juju/agents/unit-
> neutron-openvswitch-cplane-0/charm/hooks/cplane_network.py", line 127, in
> change_iface_config
> 2017-11-16 03:50:57 INFO config-changed subprocess.check_call(cmd)
> 2017-11-16 03:50:57 INFO config-changed File "/usr/lib/python2.7/subprocess.py",
> line 541, in check_call
> 2017-11-16 03:50:57 INFO config-changed raise
> CalledProcessError(retcode, cmd)
> 2017-11-16 03:50:57 INFO config-changed subprocess.CalledProcessError:
> Command '['ifup', u'eth1.10']' returned non-zero exit st
>
> When I go into the server and try to do an ifup/down manually I’m getting
> this error
> eth1.10 no such interface
>
> ubuntu@MACH4:~$ sudo -i
> root@MACH4:~# ifdown eth1.10
> ifdown: interface eth1.10 not configured
>
> but it clearly shows it is
>
> eth1.10 Link encap:Ethernet HWaddr 52:54:00:5a:57:aa
> inet6 addr: fe80::5054:ff:fe5a:57aa/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Met...

[john (g-john-p)]

adding tar of /etc/network/

[john (g-john-p)]

Attaching tar of /etc/network/* from curtin505 what works for your comparison.

[Ryan Harper (raharper)]

Thanks for the configuration files.

505 and 532 pick different vlan numbers, but I think the section that matters
most is the br-ext and it's port (eth1.XX). In 532 it's eth1.10 and in 505 it's eth1.15.

IN both cases, the config looks like this:

# Interface eth1.10
allow-br-ext eth1.10
iface eth1.10 inet manual
    <config>

# Bridge br-ext
auto br-ext
allow-ovs br-ext
iface br-ext inet static
   <config>

the br-ext port is not marked 'auto' or 'allow-auto'; which means ifupdown is going to ignore that unless you explicitly allow it like so:

ifup --allow br-ext eth1.10
ifdown --allow br-ext eth1.10

Ignore the vlan WARNING outputs (the scripts are just noisy)

root@x1:~# cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback
    dns-nameservers 10.1.0.1
    dns-search maas

auto eth0
iface eth0 inet static
    address 10.1.0.20/24
    dns-nameservers 10.1.0.1
    gateway 10.1.0.1
    mtu 1500

auto eth1
iface eth1 inet manual
    mtu 1500

auto eth2
iface eth2 inet manual
    mtu 1500

auto eth2.11
iface eth2.11 inet static
    address 192.168.64.14/24
    dns-nameservers 192.168.64.2
    mtu 1500
    vlan-raw-device eth2
    vlan_id 11

post-up route add -net 192.168.61.0 netmask 255.255.255.0 gw 192.168.64.1 metric 0 || true
pre-down route del -net 192.168.61.0 netmask 255.255.255.0 gw 192.168.64.1 metric 0 || true

# Interface eth1.10
allow-br-ext eth1.10
iface eth1.10 inet manual
    ovs_type OVSPort
    ovs_bridge br-ext
    vlan-raw-device eth1
    dns-nameservers 192.168.63.2
    mtu 9000
    bridge_name br-ext
    vlan_id 10

# Bridge br-ext
auto br-ext
allow-ovs br-ext
iface br-ext inet static
    address 192.168.63.14
    netmask 255.255.255.0
    cp_gateway 192.168.63.1
    ovs_type OVSBridge
    ovs_ports eth1.10
    bridge_ports eth1.10

root@x1:~# ifquery --list
lo
eth0
eth1
eth2
eth2.11
br-ext
root@x1:~# ifquery --list --allow br-ext
eth1.10
root@x1:~# ifdown eth1.10
ifdown: interface eth1.10 not configured
root@x1:~# ifup eth1.10 --allow br-ext
WARNING: Could not open /proc/net/vlan/config. Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
root@x1:~# ifdown eth1.10 --allow br-ext
WARNING: Could not open /proc/net/vlan/config. Maybe you need to load the 8021q module, or maybe you are not using PROCFS??
Removed VLAN -:eth1.10:-

 I've not looked at the rest of the charm code but it's possible that needs to use the --allow when using ifup/down calls.

[john (g-john-p)]

Yes we know the configurations are different. these are two different test environments. We updated one of our tests labs first and found the problems. at the time both were working under 505. I'm providing this only as a reference. The problems are not to do with these difference.

[Ryan Harper (raharper)]

On Fri, Nov 17, 2017 at 11:33 AM, john <email address hidden> wrote:

> Yes we know the configurations are different. these are two different
> test environments. We updated one of our tests labs first and found the
> problems. at the time both were working under 505. I'm providing
> this only as a reference. The problems are not to do with these
> difference.
>

Correct; in both cases, if you want to interact with any of the interfaces
that are not marked 'auto'; then your commands need to append --allow
<class>
In particular, the error reported was an interface that was part of br-ext,
so
it would need ifup --allow br-ext eth1.10

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1732202
>
> Title:
> Xenial Deploy fails when using /etc/network/interface
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1732202/+subscriptions
>

[john (g-john-p)]

that did not work.

root@MACH4:/etc/network/interfaces.d# less 50-cloud-init.cfg
root@MACH4:/etc/network/interfaces.d# ifdown --allow br-ext eth1.10
ifdown: interface eth1.10 not configured

Also, BR-EXT and ETH1.10 should have the same MAC address. This will cause problems.

[john (g-john-p)]

BR-EXT enslaves ETH1.10 within OVS. These two must have the same MAC address and they don't in the curtin 532 case. maybe the first issue will fix this issue.

root@MACH4:/etc/network/interfaces.d# ifconfig eth1.10
eth1.10 Link encap:Ethernet HWaddr 52:54:00:5a:57:aa
          inet6 addr: fe80::5054:ff:fe5a:57aa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5749 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4253 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:500579 (500.5 KB) TX bytes:324848 (324.8 KB)

root@MACH4:/etc/network/interfaces.d# ifconfig br-ext
br-ext Link encap:Ethernet HWaddr 3a:31:6b:39:16:40
          inet addr:192.168.63.14 Bcast:192.168.63.255 Mask:255.255.255.0
          inet6 addr: fe80::3831:6bff:fe39:1640/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)

[Ryan Harper (raharper)]

On Fri, Nov 17, 2017 at 1:18 PM, john <email address hidden> wrote:

> that did not work.
>
> root@MACH4:/etc/network/interfaces.d# less 50-cloud-init.cfg
> root@MACH4:/etc/network/interfaces.d# ifdown --allow br-ext eth1.10
> ifdown: interface eth1.10 not configured
>

that's because it's not up (from an ifupdown perspective; it keeps state in
/run/network/);

ifup --allow br-ext eth1.10; then you can down it.

It's not clear to me (without more reading of your code) why it needs to go
down.
MTU changes do not require up/downing of interfaces.

>
> Also, BR-EXT and ETH1.10 should have the same MAC address. This will
> cause problems.
>

Let me look at the configs.

There are no hwaddress configurations in the eni file, so I suspect that
something else is going on w.r.t the br-ext bridge configuration.

The current state of /etc/network/interfaces.d/50-cloud-init.cfg
isn't complete (it's missing the tun and the br-tun) configuration.
It appears to me that the charm configuration is iterative, adding one
change at a time; and if during one step, the ifup/ifdown command fails
then it likely prevents the rest of the config from being generated and
applied.

I don't have ovs_bridge installed, but just using regular linux bridge
I put in bridge_ports eth1.10 in the br-ext configuration, and I see when
br-ext comes up, it creates eth1.10 vlan, and adds to the br-ext bridge
at which time the bridge gets the eth1.10 mac address.

root@x1:~# ifdown -v br-ext
Parsing file /etc/network/interfaces.d/50-cloud-init.cfg
Configuring interface br-ext=br-ext (inet)
/bin/run-parts --verbose /etc/network/if-down.d
run-parts: executing /etc/network/if-down.d/resolvconf
run-parts: executing /etc/network/if-down.d/upstart

/bin/ip addr del 192.168.63.14/255.255.255.0 broadcast 192.168.63.255 dev
br-ext label br-ext
/bin/ip link set dev br-ext down
/bin/run-parts --verbose /etc/network/if-post-down.d
run-parts: executing /etc/network/if-post-down.d/bridge
WARNING: Could not open /proc/net/vlan/config. Maybe you need to load the
8021q module, or maybe you are not using PROCFS??
Removed VLAN -:eth1.10:-
run-parts: executing /etc/network/if-post-down.d/ifenslave
+ [ inet = meta ]
+ BOND_PARAMS=/sys/class/net/br-ext/bonding
+ [ -f /sys/class/net/br-ext/master/bonding/slaves ]
+ [ ! -f /sys/class/net/br-ext/bonding/slaves ]
+ exit
run-parts: executing /etc/network/if-post-down.d/vlan
root@x1:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:16:3e:17:e8:1a
          inet addr:10.1.0.20 Bcast:10.1.0.255 Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe17:e81a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1094 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:163351 (163.3 KB) TX bytes:1332 (1.3 KB)

eth1 Link encap:Ethernet HWaddr 00:16:3e:c0:eb:ff
          inet6 addr: fe80::216:3eff:fec0:ebff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:1085 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31 errors:0 dropped:0 overruns...

[john (g-john-p)]

Ryan, is it possible to do a webex/zoom into my environment. I know you are giving your thoughts on what might be wrong. What I can tell you is this works for curtin505 and not for 532, so there is something fundatmentally wrong with the config of these machines. I don't know enough about what you changed to debug it. maybe if you can log onto our system you can see. I'm very concerned that you didn't test with OVS or event your openstack environment because it should behave the same way. OVS will enslave a real interface and that it needs to be the same mac address. the fact that the problem is with the OVS interface mean it might be some incompatibilty with 538 and OVS.

Can we schedule a call?

[john (g-john-p)]

more info:
please see the message:
RTNETLINK answers: Invalid argument
RTNETLINK answers: Numerical result out of range
Failed to bring up eth1.10.

root@MACH4:~# ifup -v br-ext
Configuring interface br-ext=br-ext (inet)
/bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/bridge
run-parts: executing /etc/network/if-pre-up.d/ethtool
run-parts: executing /etc/network/if-pre-up.d/ifenslave
+ [ inet = meta ]
+ IF_BOND_SLAVES=
+ [ ]
+ [ ]
+ [ -z ]
+ exit
run-parts: executing /etc/network/if-pre-up.d/mtuipv6
run-parts: executing /etc/network/if-pre-up.d/openvswitch

SIOCSIFFLAGS: Network is down
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
RTNETLINK answers: Invalid argument
RTNETLINK answers: Numerical result out of range
Failed to bring up eth1.10.
run-parts: executing /etc/network/if-pre-up.d/vlan
/bin/ip addr add 192.168.63.14/255.255.255.0 broadcast 192.168.63.255 dev br-ext label br-ext
/bin/ip link set dev br-ext up

/bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/000resolvconf
run-parts: executing /etc/network/if-up.d/ethtool
run-parts: executing /etc/network/if-up.d/ifenslave
+ [ inet = meta ]
+ [ ]
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/mtuipv6
run-parts: executing /etc/network/if-up.d/openssh-server
run-parts: executing /etc/network/if-up.d/upstart

[john (g-john-p)]

Ryan - I think somehow curtin532 has exposed this bug https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1567744

[David Britton (dpb)]

Marked Incomplete. We suspect this is a problem in that charm that needs to be fixed, but to be sure, please attach:

/etc/network/interfaces
/etc/network/interfaces.d/*

*before* the charm in question is executed. Usually, it would be enough to deploy to the same node, but use the 'ubuntu' charm to be able to capture these things.

Thanks!

[john (g-john-p)]

please close. We suspect that there was an old version of the xenial image or corrupted apt-update. We rebuilt our environment and it now works.