Cloud-init update renders secondary addresses to be incompatible with standard tools

[Ben Howard]

The change of how cloud-init renders /etc/network/interface.d/50-cloud-init.cfg, standard tools no longer work as expected:
* resolvconf will nullify nameservers
* if* commands ignore secondary addresses

[ORIGINAL REPORT]

Regresion from Bug #1657940.

When provisioning with multiple eth0 addresses, /etc/resolv.conf is empty:

Consider:
root@tester:~# cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 138.197.98.102
    dns-nameservers 8.8.8.8 8.8.4.4
    gateway 138.197.96.1
    netmask 255.255.240.0

# control-alias eth0
iface eth0 inet static
    address 10.17.0.11
    netmask 255.255.0.0

Which then yields an empty /etc/resolv.conf:
root@tester:/run/resolvconf# cat interface/eth0.inet

root@tester:/run/resolvconf# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

The problem is that resolvconfg does pattern matching for eth*.inet. The second definition of eth0 has no nameserver and therefore overrides the definition.

[Ryan Harper (raharper)]

The network config in the previous bug used DHCP on v4 and v6; it appears this instance has a different network config injected. Can you attach the network configuration?

As a workaround, you can include a nameserver configuration

- type: nameserver:
  address:
    - 8.8.8.8
    - 8.8.8.3

Or also include the dns-nameservers attribute in the second subnet configuration.

This will render dns-nameservers entry under lo
which will get picked up.

Given the above config, it looks like the ifupdown resolveconf hooks are not multi-iface-stanza aware.

[Ben Howard (darkmuggle-deactivatedaccount)]

The following is the network_config:
'_network_config': {'config': [{'mac_address': 'ca:22:2d:f4:1a:5e',
                                 'name': 'eth0',
                                 'subnets': [{'address': '138.197.98.102',
                                              'control': 'auto',
                                              'dns_nameservers': ['8.8.8.8',
                                                                  '8.8.4.4'],
                                              'gateway': '138.197.96.1',
                                              'netmask': '255.255.240.0',
                                              'type': 'static'},
                                             {'address': '10.17.0.11',
                                              'control': 'auto',
                                              'netmask': '255.255.0.0',
                                              'type': 'static'}],
                                 'type': 'physical'}],
                     'version': 1},

That was extracted from /var/lib/cloud/obj.pkl.

[Ben Howard (darkmuggle-deactivatedaccount)]

Changing the cloud-metadata won't work in this case, since the the code for the DigitalOcean metadata source is assigning the Nameservers to the primary interface. Also, adding the DNS resolvers to the loopback interfaces doesn't work because the loopback interface is already up before the definition is added.

[Ben Howard (darkmuggle-deactivatedaccount)]

Patch submitted to Cloud-init via:
https://code.launchpad.net/~utlemming/cloud-init/+git/cloud-init-1/+merge/321001

[Ben Howard (darkmuggle-deactivatedaccount)]

The patch in comment #4 moves the revolvers to the loopback interface; resolve conf picks it up.

[Ben Howard (darkmuggle-deactivatedaccount)]

I would like to nominate this bug for the Xenial, Yakkety and Zesty branches as well.

[Ben Howard (darkmuggle-deactivatedaccount)]

So while my patch for the DigitalOcean datasource fixes this usecase, the rendering of additional addresses actually breaks all the "if*" tools for secondary addresses and even IPv6. Consider:

root@utl-ubuntu-16-04-x64-50307:~# ifquery eth0
address: 138.197.91.79
gateway: 138.197.80.1
netmask: 255.255.240.0
broadcast: 138.197.95.255
root@utl-ubuntu-16-04-x64-50307:~# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 4a:41:c5:1b:8a:a0 brd ff:ff:ff:ff:ff:ff
    inet 138.197.91.79/20 brd 138.197.95.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.17.0.6/16 brd 10.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2604:a880:800:10::2ced:3001/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::4841:c5ff:fe1b:8aa0/64 scope link
       valid_lft forever preferred_lft forever

[Ben Howard (darkmuggle-deactivatedaccount)]

Additionally, "ifconfig" no longer shows secondary addresses. Notice that `10.17.0.6/16` in #7 above is not defined:

root@utl-ubuntu-16-04-x64-50307:~# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 4a:41:c5:1b:8a:a0
          inet addr:138.197.91.79 Bcast:138.197.95.255 Mask:255.255.240.0
          inet6 addr: fe80::4841:c5ff:fe1b:8aa0/64 Scope:Link
          inet6 addr: 2604:a880:800:10::2ced:3001/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:844 errors:0 dropped:0 overruns:0 frame:0
          TX packets:653 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:107141 (107.1 KB) TX bytes:91481 (91.4 KB)

[Ben Howard (darkmuggle-deactivatedaccount)]

I've withdrawn my PR for the DigitalOcean specific fix after an offline conversation.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2842