ENI format - configuring metadata network-interfaces with iface defined twice but different family breaks all networking on cloud-init instance

[Andreas Karis]

Description of problem:
Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance. This only works when configuring a sub interface.

The following breaks networking:
~~~
auto eth2
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
iface eth2 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
+++
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
+++

And https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html
+++
Options are usually indented for clarity (as in the example above) but are not required to be.
+++

The following works correctly:
~~~
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
auto eth2:0
iface eth2:0 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

Additional info:
result working:

Working instance: ./sosreport-20180125-075718/svc-1-lvsrouter/var/log/cloud-init-output.log
~~~
Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 07:48:44 +0000. Up 12.41 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 07:48:47 +0000. Up 16.00 seconds.
ci-info: +++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo: | True | . | . | d | . |
ci-info: | eth1: | True | 10.250.246.171 | 255.255.252.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth1: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth2: | True | 192.168.0.166 | 255.255.0.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth2: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | 10.247.246.174 | 255.255.252.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth3: | True | 172.16.30.226 | 255.255.255.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth3: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 192.168.0.1 | 0.0.0.0 | eth2 | UG |
ci-info: | 1 | 10.247.244.0 | 0.0.0.0 | 255.255.252.0 | eth0 | U |
ci-info: | 2 | 10.250.244.0 | 0.0.0.0 | 255.255.252.0 | eth1 | U |
ci-info: | 3 | 172.16.30.0 | 0.0.0.0 | 255.255.255.0 | eth3 | U |
ci-info: | 4 | 192.168.0.0 | 0.0.0.0 | 255.255.0.0 | eth2 | U |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 07:48:50 +0000. Up 19.03 seconds.
~~~

result not working:
~~~
Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 11:50:02 +0000. Up 7.98 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 11:50:10 +0000. Up 15.71 seconds.
2018-01-25 06:50:10,348 - util.py[WARNING]: Route info failed: Unexpected error while running command.
Command: ['netstat', '-rn']
Exit code: 1
Reason: -
Stdout: Kernel IP routing table
        Destination Gateway Genmask Flags MSS Window irtt Iface
Stderr: -
ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo: | True | . | . | d | . |
ci-info: | eth1: | True | . | . | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth2: | True | . | . | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | . | . | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth3: | True | . | . | . | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 11:50:12 +0000. Up 17.90 seconds.
~~~

[Andreas (andreas-redhat-bugs)]

Description of problem:
Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance. This only works when configuring a sub interface.

The following breaks networking:
~~~
auto eth2
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
iface eth2 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
+++
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
+++

And https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html
+++
Options are usually indented for clarity (as in the example above) but are not required to be.
+++

The following works correctly:
~~~
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
auto eth2:0
iface eth2:0 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb
~~~

Additional info:
result working:

Working instance: ./sosreport-20180125-075718/svc-1-lvsrouter/var/log/cloud-init-output.log
~~~
Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 07:48:44 +0000. Up 12.41 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 07:48:47 +0000. Up 16.00 seconds.
ci-info: +++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo: | True | . | . | d | . |
ci-info: | eth1: | True | 10.250.246.171 | 255.255.252.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth1: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth2: | True | 192.168.0.166 | 255.255.0.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth2: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | 10.247.246.174 | 255.255.252.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth0: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: | eth3: | True | 172.16.30.226 | 255.255.255.0 | . | xx:xx:xx:xx:xx:xx |
ci-info: | eth3: | True | . | . | d | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+----...

[Andreas (andreas-redhat-bugs)]

Tested with cloud-init-0.7.9-20.el7.x86_64

[Ryan (ryan-redhat-bugs)]

I don't believe this is intended to work. Did defining the interface twice previously work for the customer?

The current upstream release explicitly errors out when you do this with a parse error.

[Andreas (andreas-redhat-bugs)]

Hi,

cloud-init upstrea in cloudinit/net/eni.py
https://github.com/number5/cloud-init/blob/master/cloudinit/net/eni.py#L193
~~~
 elif option == "iface":
            iface, family, method = split[1:4]
            if iface not in ifaces:
                ifaces[iface] = {
                    # Include the source path this interface was found in.
                    "_source_path": src_path
                }
            elif 'family' in ifaces[iface]:
                raise ParserError(
                    "Interface %s can only be defined once. "
                    "Re-defined in '%s'." % (iface, src_path))
            ifaces[iface]['family'] = family
            ifaces[iface]['method'] = method
            currif = iface
~~~

a) iface eth2 inet static
b) iface eth2 inet6 static

a)
iface = eth2
family = inet
method = static

b)
iface = eth2
family = inet6
method = static

inet will define ifaces['eth2']['family'] as 'inet' and then on the next pass with inet6, `elif 'family' in ifaces[iface]` will evaluate to true and this throw an error.

The actual error message doesn't seem to figure in any of the collected customer data, but from the above, it seems pretty clear that the ENI format accepts only one interface and not different tuples of iface/family.

[Andreas (andreas-redhat-bugs)]

I'm bringing this upstream ...

http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni

states that:
~~~
Please reference existing documentation for the /etc/network/interfaces(5) format.
~~~

From https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html - the man page doesn't seem to say anything else about duplicate iface declarations with different families:
~~~
Stanzas defining logical interfaces start with a line consisting of the word "iface" followed by the name of the logical interface. In simple configurations without mapping stanzas this name should simply be the name of the physical interface to which it is to be applied. (The default mapping script is, in effect, the echo command.) The interface name is followed by the name of the address family that the interface uses. This will be "inet" for TCP/IP networking, but there is also some support for IPX networking ("ipx"), and IPv6 networking ("inet6"). Following that is the name of the method used to configure the interface.
(...)
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
~~~
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
~~~

[Andreas Karis (akaris)]

I'm bringing this upstream ...

http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni

states that:
~~~
Please reference existing documentation for the /etc/network/interfaces(5) format.
~~~

From https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html - the man page doesn't seem to say anything else about duplicate iface declarations with different families:
~~~
Stanzas defining logical interfaces start with a line consisting of the word "iface" followed by the name of the logical interface. In simple configurations without mapping stanzas this name should simply be the name of the physical interface to which it is to be applied. (The default mapping script is, in effect, the echo command.) The interface name is followed by the name of the address family that the interface uses. This will be "inet" for TCP/IP networking, but there is also some support for IPX networking ("ipx"), and IPv6 networking ("inet6"). Following that is the name of the method used to configure the interface.
(...)
~~~

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
~~~
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef
~~~

[Andreas Karis (akaris)]

The implementation doesn't seem to like this, though:

cloud-init upstrea in cloudinit/net/eni.py
https://github.com/number5/cloud-init/blob/master/cloudinit/net/eni.py#L193
~~~
 elif option == "iface":
            iface, family, method = split[1:4]
            if iface not in ifaces:
                ifaces[iface] = {
                    # Include the source path this interface was found in.
                    "_source_path": src_path
                }
            elif 'family' in ifaces[iface]:
                raise ParserError(
                    "Interface %s can only be defined once. "
                    "Re-defined in '%s'." % (iface, src_path))
            ifaces[iface]['family'] = family
            ifaces[iface]['method'] = method
            currif = iface
~~~

a) iface eth2 inet static
b) iface eth2 inet6 static

a)
iface = eth2
family = inet
method = static

b)
iface = eth2
family = inet6
method = static

inet will define ifaces['eth2']['family'] as 'inet' and then on the next pass with inet6, `elif 'family' in ifaces[iface]` will evaluate to true and this throw an error.

The actual error message doesn't seem to figure in any of the collected customer data, but from the above, it seems pretty clear that the ENI format accepts only one interface and not different tuples of iface/family.

[Ryan Harper (raharper)]

Thanks for filing the bug. The ifupdown eni parser in eni.py does not handle reading multiple stanzas of the same interface; however it _is_ support in eni; so this is a bug in the eni format parser.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3105