lxd containers try to DHCP on eth0 even when we know it won't work

[Dan Watkins]

The lxd_metadata scripts write out templates that produce a /etc/network/interfaces.d/eth0.cfg file.

When operating in link-local mode (i.e. without IPv4 access on eth0), this is written out as:

    # The primary network interface
    auto eth0
    iface eth0 inet manual

When operating in any other mode (i.e. with a bridge), it's written out as that with s/manual/dhcp/.

Unfortunately, cloud-init doesn't work out what's going on here properly, so it writes out ENI.d/50-cloud-init.cfg which _always_ configures eth0 to use DHCP.

For whatever reason, this is the eth0 configuration that the system ends up using. This means that cloud-init doesn't complete in the container until networking.service gives up on doing DHCP (which takes about 5 minutes).

[Dan Watkins (oddbloke)]

So I think the way to fix this is to remove the eth0.cfg template for xenial and forwards, and replace it with cloud-init networking configuration.

The template that lxd currently uses for eth0.cfg is http://paste.ubuntu.com/15963686/. I believe that this template can be replaced with:

version: 1
network:
    - type: physical
      name: eth0
      subnets:{% if config_get("user.network_mode", "") == "link-local" %}
        - type: manual{% else %}
        - type: dhcp
          control: auto{% endif %}

as a template for /var/lib/cloud/seed/nocloud-net/network-config.

stgraber thinks this looks good, so I've pushed it over to smoser/rharper to get their input.