Support static network configuration even on already configured devices

[Vlastimil Holer]

Some datasources (e.g. OpenNebula) support full static network configuration. It's done in local execution phase by pushing new interfaces configuration to *distro.apply_network*. This new configuration is written on disk and activated by calling ifup on particular devices. Unfortunatelly it can't be guaranteed that full local phase is executed before any network configuration is done by system. Mentioned steps are OK only for devices not present in former network configuration or not configurated to start on boot (e.g. eth0 configured on boot to take address from DHCP, the new static configuration is not applied on this device, it's already up and ifup just passes).

It would be good to first put interfaces down before writing new configuration.

[Scott Moser (smoser)]

Currently networking coming up happens independent of cloud-init. We could potentially make cloud-init local block networking from coming up, but I don't think thats something I'd want to take on for cloud-init 0.7.3 / Ubuntu 13.10.

I'm not really sure how much a hack this is, if at all.
The solution I'd propose would be:
 * use 'ifquery' to get the list of networking interfaces in the original
   from cloudinit import util
   (out, _err) = util.subp(['ifquery', '--list', '--allow', 'auto'])
   orig_list = out.splitlines()

 * use ifquery to get the list of 'auto' interfaces in your new list:
   (out, _err) = util.subp(['ifquery', '--interfaces=/tmp/my.tmp.interfaces',
                            --list', '--allow', 'auto'])
   new_list = out.splitlines()

 * take down interfaces in orig_list and new_list
   for iface in [i for i in orig_list if i in new_list]:
      util.subp(['ifdown', iface]

   # have to ignore errors above in case the interface wasn't up.

 * write/update /etc/network/interfaces file

 * bring all interfaces up:
   util.subp(['ifup', '-a'])

I opened bug 1226067 (http://pad.lv/1226067) to address an issue where
ifquery is busted in saucy, but that will get fixed.

[Vlastimil Holer (vlastimil-holer)]

Scott, don't take it bad, but I don't like the solution with ifquery. I believe it's Ubuntu/Debian specific and it would be fine to have a more platform independent solution. At least for those distros currently supported by cloud-init.

[Scott Moser (smoser)]

On Mon, 16 Sep 2013, Vlastimil Holer wrote:

> Scott, don't take it bad, but I don't like the solution with ifquery. I
> believe it's Ubuntu/Debian specific and it would be fine to have a more
> platform independent solution. At least for those distros currently
> supported by cloud-init.

I'm not opposed to something at 'distro' level that would do the same
thing. I actually had thought that this datasource was currently
ubuntu/debian specific, but realize just now why using
'network-interfaces' is not.

That doesn't really change anything though, just where the code would be
put, and the fact that you'll have to do it for RH too.

Ie, you'll make the changes to bring down interfaces first in
 cloudinit/distros
rather than in the datasource.

Scott

[Vlastimil Holer (vlastimil-holer)]

I have spent a little time and prepared this:
http://bazaar.launchpad.net/~vlastimil-holer/cloud-init/net-reconfigure/revision/870
(just as an working example, tested on Debian so far)

I don't know if you'll like...

[Josh Boon (josh-boon)]

The new interfaces.d in trusty makes this even more fun. I've been using local as datasource and user-data to configure and the introduction of interfaces.d completely breaks the old kinda working network config.

[Daryl Robbins (darylrobbins)]

Is there any work around for this issue? Thanks!

[Vlastimil Holer (vlastimil-holer)]

Yes, reboot instance after initial configuration. Put this into your
cloud-init configuration:

power_state:
  mode: reboot
  message: Initial configuration done by cloud-init, forcing reboot to
apply changes.

[Javi Fontan (jfontan)]

I managed to get it working setting all the interfaces down with this snippet:

bootcmd:
  - ifdown -a

[Daryl Robbins (darylrobbins)]

Thanks for the quick responses! My issue is that I'm trying to run through additional steps after the network is configured: namely download and run chef-client. (on trusty Ubuntu 14.04)

Sometimes, the timings work out but most of the time, Ubuntu's network config and cloud-init step on each other's toes, causing the Chef omnibus install to fail.

chef:
  install_type: omnibus
  ...
runcmd:
  - ifdown eth0
  - ifup eth0
  - ifup eth1
  - chef-client

Is there a potential way to get it working which plays nicely with running additional init steps? Thanks so much!

[Josh Boon (josh-boon)]

If you're using the cloud images in your own network I'd suggest just modifying the images and removing the eth0 config from /etc/interfaces.d Once I did that I had no issue applying static for my interfaces.

[Daryl Robbins (darylrobbins)]

Thanks, Josh. I removed the eth0 config file from the image, but am seeing a 'Route Info Failed'. Did you have to do anything else special to get it to work? I tried 'ifup -a' as both a bootcmd and runcmd but to no avail.

[Josh Boon (josh-boon)]

Here's what I have in meta-data:

#cloud-config
network-interfaces: |
  auto eth0
  iface eth0 inet static
  address 192.168.100.55
  network 192.168.100.0
  netmask 255.255.255.0
  broadcast 192.168.100.255
  gateway 192.168.100.1
  dns-nameservers $DNS
  dns-search $DOMAIN

You should be able to drop the DNS if you're managing that elsewhere.

[Daryl Robbins (darylrobbins)]

Ugh, it just took me half an hour to notice that my only problem after removing eth0 from the image was that I forgot the auto. Thanks!

[Ken Schroeder (kschroed)]

Using runcmd helps with some scenarios but lot of things built into cloud-init already like package installs as well won't work if networking isn't online. I'm having similar challenges due to the start up ordering and having to build a lot of the functions cloud-init already servers in runcmd section after restarting networking once the static ip configs have been laid down.

[Kenneth Burger (burgerk)]

Following with the thought process of the proposal by smoser in #3, I have made the following change in cloudinit/distros/__init__.py to work around this in my environment. ( doesn't go through the complexity of taking down only the auto interfaces )

def apply_network(self, settings, bring_up=True):
        # Write it out
        dev_names = self._write_network(settings)
        # Now try to bring them up
        if bring_up:
            self._bring_down_interfaces(dev_names) <---- new method
            return self._bring_up_interfaces(dev_names)
        return False

Then added these default implementations:

    def _bring_down_interface(self, device_name):
        cmd = ['ifdown', device_name]
        LOG.debug("Attempting to run bring down interface %s using command %s",
                   device_name, cmd)
        try:
            (_out, err) = util.subp(cmd)
            if len(err):
                LOG.warn("Running %s resulted in stderr output: %s", cmd, err)
            return True
        except util.ProcessExecutionError:
            util.logexc(LOG, "Running interface command %s failed", cmd)
            return False

    def _bring_down_interfaces(self, device_names):
        am_failed = 0
        for d in device_names:
            if not self._bring_down_interface(d):
                am_failed += 1
        if am_failed == 0:
            return True
        return False

Since Ubuntu always returns --all for device_names, so implement in debian.py as:

    def _bring_down_interfaces(self, device_names):
        use_all = False
        for d in device_names:
            if d == 'all':
                use_all = True
        if use_all:
            return distros.Distro._bring_down_interface(self, '--all')
        else:
            return distros.Distro._bring_down_interfaces(self, device_names)

[Kenneth Burger (burgerk)]

Just noticed the duplicate Bug #1275098 ... The approach there looks very similar to what I did.

[Ahmed Rahal (arahal)]

As discussed with smoser on irc, putting that code in __init__.py is not acceptable.
As soon as I get the chance, I'll put it in the distro-specific code and re-propose. Am quite busy ATM :(

[Madhu Pavan (kmp)]

Hi arahal,
As one of the functionalities of __init__.py is to allows you to define methods at package level, which are common for all the distros in our case. Why don't we patch __init__.py as the issue is common for all the distros and patching it will solve the bug. Can you please share the details why smoser don't want the patch in __init__.py and rather have it in distro specific files? Thanks.

[Ahmed Rahal (arahal)]

Hi Madhu,

Here you have the full text of our chat on #cloud-init
http://paste.openstack.org/show/135506/

I still am planning to port https://bugs.launchpad.net/cloud-init/+bug/1275098 so it becomes acceptable.
Need to gather some sleep first ;)

[Scott Moser (smoser)]

I'm going to mark this fix-released.
The general bug as described in the description is that cloud-init can't correctly apply networking for all interfaces.

cloud-init local applies networking configuration to the system, and should apply before the system brings networking up. Thus appearing to the system as if the networkign config was already there, and should be brought up properly as it would be on reboot.

If you think this bug is not fixed, its probably best to file a new bug, describe the problem, and attach output of 'cloud-init collect-logs'.