A network bridge for containers does not get created if MAAS interface aliases are used

[Dmitrii Shcherbakov]

I have a poor man's network spaces test setup where I only have a single VLAN available for eth1 interface and different subnets belong to different networks spaces on a single fabric on a single VLAN.

A part of a switchport config:

Eth126/1/1 pillan:eth0 connected 2594 full a-1000
Eth126/1/37 pillan:eth1 connected 2595 full a-1000

Converting switchports to trunks which allow multiple vlans is in works (so that I can create a bunch of VLAN interfaces) but I decided to see how Juju 2.1.2 handles interface aliases that MAAS (2.1.5) creates.

MAAS setup looks as follows:

Name Type Fabric VLAN SubnetIP Address
eth1 Physical isolated untagged 10.232.16.0/21 (public-subnet) 10.232.16.5 (Auto assign)
eth1:1 Alias isolated untagged 10.232.24.0/21 (ceph-access-subnet) 10.232.24.5 (Auto assign)
eth1:2 Alias isolated untagged 10.232.32.0/21 (ceph-replica-subnet) 10.232.32.5 (Auto assign)
eth1:3 Alias isolated untagged 10.232.40.0/21 (admin-subnet) 10.232.40.5 (Auto assign)
eth1:4 Alias isolated untagged 172.16.10.0/24 (internal-subnet) 172.16.10.5 (Auto assign)

Fabric VLAN Subnet Available IPs Space
isolated untagged 10.232.16.0/21 (public-subnet) 98% public-space
                                10.232.24.0/21 (ceph-access-subnet) 100% ceph-access-space
                                10.232.32.0/21 (ceph-replica-subnet) 100% ceph-replica-space
                                10.232.40.0/21 (admin-subnet) 100% admin-space
                                172.16.10.0/24 (internal-subnet) 96% internal-space

On the actual node deployment, MAAS simply assigns different addresses to the same interface (which is perfectly valid).

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 78:e7:d1:24:cf:c1 brd ff:ff:ff:ff:ff:ff
    inet 10.232.16.1/21 brd 10.232.23.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.232.24.1/21 brd 10.232.31.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.232.32.1/21 brd 10.232.39.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.232.40.1/21 brd 10.232.47.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 172.16.10.1/24 brd 172.16.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::7ae7:d1ff:fe24:cfc1/64 scope link
       valid_lft forever preferred_lft forever

A bridge for containers simply doesn't get created in this scenario.

I understand that this is quite a useless scenario in real life but I was wondering if this has been considered at all and if it is a bug or a "won't fix".

Seems to me that this could be done by plugging multiple container interfaces into a single bridge though I realize that there are probably corner cases making it hard to implement.

[John A Meinel (jameinel)]

Is it that no bridge is created at all for eth1. Can you include the contents of /etc/network/interfaces after you've configured the host machine in this way?
Can you run debug level with juju (juju model-config -m controller logging-config="<root>=DEBUG", and the same for the active model) and include the content of the logs for the controller machine (and for the machine that is creating the containers)?
Some of the information is worked out by the controller and logged there, and some is worked out on the machine, hence the need for both log files.
I'm wondering if we're:
1) Unable to figure out what device to bridge
2) Find the device, but failing to create the bridge
3) Failing to assign the right ip addresses to the bridge,
etc.

I don't think we've done a lot of work around multiple subnets-per-device so there are a few places it could fail.
Offhand, I'm not sure the *utility* of not having any actual separation of traffic (vlans give you the ability to configure visibility at the switch layer, etc).

So there are aspects of "its not worth modeling this", but it may be that we're actually missing something we really do want, hence the request for more info.

[Dmitrii Shcherbakov (dmitriis)]

I don't have the env anymore but as for /etc/network/interfaces, it can be seen from the `ip a s` output I pasted above that eth1 has multiple interfaces configured and there are no separate interfaces for each subnet (which makes sense kernel-wise).

I will try to replicate this scenario in a smaller lab (probably virtual) as the one I used will be used for demos so I switched to Juju 2.2 and MAAS 2.2 there and I will use the new network space concept and VLANs.

So I think it might make sense to reproduce it on Juju 2.2 instead.

Btw, on Juju 2.1.2 and MAAS 2.1.5 while working with a scenario where one physical interface has multiple VLAN virtual interfaces (each subnet is related to a different network space), I get an interesting issue:

machine-3: 17:31:40 ERROR juju.provisioner cannot start instance for machine "3/lxd/2": unexpected number of interfaces in response from creating device: [0xc821a19930 0xc821a19a00]

This error message comes from the fact that gomaasapi returns multiple interfaces (which is true as for vlans there are multiple virtual interfaces with the same MAC):

ᚠ 2.1 provider/maas/environ.go

func (env *maasEnviron) allocateContainerAddresses2(hostInstanceID instance.Id, containerTag names.MachineTag, preparedInfo []network.InterfaceInfo) ([]network.InterfaceInfo, error) {
...

 interface_set := device.InterfaceSet()
 if len(interface_set) != 1 {
  // Shouldn't be possible as machine.CreateDevice always returns us
  // one interface.
  return nil, errors.Errorf("unexpected number of interfaces in response from creating device: %v", interface_set)
 }
 primaryNICVLAN := interface_set[0].VLAN()

--
goomaasapi

// InterfaceSet implements Device.
func (d *device) InterfaceSet() []Interface {
 result := make([]Interface, len(d.interfaceSet))
 for i, v := range d.interfaceSet {
  v.controller = d.controller
  result[i] = v
 }
 return result
}

---

ubuntu@empusa:~$ brctl show
bridge name bridge id STP enabled interfaces
br-e0 8000.78e7d124e2e0 no e0
br-e1.2729 8000.78e7d124e2e1 no e1.2729
br-e1.2732 8000.78e7d124e2e1 no e1.2732

--

This comes from the fact that interface_set in the `maas <name> machines read` will return multiple interfaces per Juju's request.

So I don't think juju 2.1.x is usable in any of the two scenarios - either with aliases or with VLANs at this point.

[John A Meinel (jameinel)]

What you're mentioning about multiple interfaces is a bug in the retry
logic that was implemented for a different bug. We should open a direct bug
on that and get it fixed for 2.2.

On Wed, Apr 12, 2017 at 6:50 PM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> I don't have the env anymore but as for /etc/network/interfaces, it can
> be seen from the `ip a s` output I pasted above that eth1 has multiple
> interfaces configured and there are no separate interfaces for each
> subnet (which makes sense kernel-wise).
>
>
> I will try to replicate this scenario in a smaller lab (probably virtual)
> as the one I used will be used for demos so I switched to Juju 2.2 and MAAS
> 2.2 there and I will use the new network space concept and VLANs.
>
> So I think it might make sense to reproduce it on Juju 2.2 instead.
>
> Btw, on Juju 2.1.2 and MAAS 2.1.5 while working with a scenario where
> one physical interface has multiple VLAN virtual interfaces (each subnet
> is related to a different network space), I get an interesting issue:
>
> machine-3: 17:31:40 ERROR juju.provisioner cannot start instance for
> machine "3/lxd/2": unexpected number of interfaces in response from
> creating device: [0xc821a19930 0xc821a19a00]
>
> This error message comes from the fact that gomaasapi returns multiple
> interfaces (which is true as for vlans there are multiple virtual
> interfaces with the same MAC):
>
> ᚠ 2.1 provider/maas/environ.go
>
> func (env *maasEnviron) allocateContainerAddresses2(hostInstanceID
> instance.Id, containerTag names.MachineTag, preparedInfo
> []network.InterfaceInfo) ([]network.InterfaceInfo, error) {
> ...
>
> interface_set := device.InterfaceSet()
> if len(interface_set) != 1 {
> // Shouldn't be possible as machine.CreateDevice always
> returns us
> // one interface.
> return nil, errors.Errorf("unexpected number of interfaces
> in response from creating device: %v", interface_set)
> }
> primaryNICVLAN := interface_set[0].VLAN()
>
>
> --
> goomaasapi
>
> // InterfaceSet implements Device.
> func (d *device) InterfaceSet() []Interface {
> result := make([]Interface, len(d.interfaceSet))
> for i, v := range d.interfaceSet {
> v.controller = d.controller
> result[i] = v
> }
> return result
> }
>
> ---
>
> ubuntu@empusa:~$ brctl show
> bridge name bridge id STP enabled interfaces
> br-e0 8000.78e7d124e2e0 no e0
> br-e1.2729 8000.78e7d124e2e1 no e1.2729
> br-e1.2732 8000.78e7d124e2e1 no e1.2732
>
> --
>
> This comes from the fact that interface_set in the `maas <name> machines
> read` will return multiple interfaces per Juju's request.
>
> So I don't think juju 2.1.x is usable in any of the two scenarios -
> either with aliases or with VLANs at this point.
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1681902
>
> Title:
> A network bridge for containers does not get crea...

[Tytus Kurek (tkurek)]

https://bugs.launchpad.net/juju/+bug/1690805

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]