cloud-init generates a traceback if a default route already exists during ephemeral network setup

[Robert Schweikert]

If a route already exists when the ephemeral network exists cloud-init will generate the following traceback:

2020-01-16 21:14:22,584 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceOracle.DataSourceOracle'> failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 760, in find_source
    if s.update_metadata([EventType.BOOT_NEW_INSTANCE]):
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 649, in update_metadata
    result = self.get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/__init__.py", line 273, in get_data
    return_value = self._get_data()
  File "/usr/lib/python2.7/site-packages/cloudinit/sources/DataSourceOracle.py", line 195, in _get_data
    with dhcp.EphemeralDHCPv4(net.find_fallback_nic()):
  File "/usr/lib/python2.7/site-packages/cloudinit/net/dhcp.py", line 57, in __enter__
    return self.obtain_lease()
  File "/usr/lib/python2.7/site-packages/cloudinit/net/dhcp.py", line 109, in obtain_lease
    ephipv4.__enter__()
  File "/usr/lib/python2.7/site-packages/cloudinit/net/__init__.py", line 920, in __enter__
    self._bringup_static_routes()
  File "/usr/lib/python2.7/site-packages/cloudinit/net/__init__.py", line 974, in _bringup_static_routes
    ['dev', self.interface], capture=True)
  File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 2083, in subp
    cmd=args)
ProcessExecutionError: Unexpected error while running command.

This is a regression from 19.1 on SUSE where exiting routes were simply skipped.

[Robert Schweikert (rjschwei)]

https://github.com/canonical/cloud-init/pull/174

[Ryan Harper (raharper)]

Can you capture cloud-init collect-logs ? In Oracle, I suspect this is related to iscsi root where initramfs already has networking up; in Ubuntu we're collecting the existing configuration from the initramfs as a network-config source and we don't bring up Ephemeral DHCP to crawl IMDS; I wonder if that's missing on SuSE path (knowing whether networking is already up due to initramfs/iscsiroot)?

[Robert Schweikert (rjschwei)]

Yes, this is in OCI.

I am not in a position to run cloud-init collect-logs as I am not able to get into a system with cloud-init 19.4 just yet.

[Robert Schweikert (rjschwei)]

Whatever the code in net/cmdline does is certainly very distribution specific as we all decided collectively/separately to do things differently. It is not really a surprise that the detection that there is already a network from booting off iscsi is not working.

Thinking there should be a distribution independent way to figure out if we already have a network connection or not.

[Ryan Harper (raharper)]

Unfortunately each distro tends to have their own initramfs networking config format. As such, cloudinit/net/cmdline.py has implemented klibc parsing (which Ubuntu/Debian support), but dracut does something different; and I'm not sure what SuSE does here; but adding a parser for the initramfs format used would handle this.

https://github.com/canonical/cloud-init/blob/master/cloudinit/net/cmdline.py#L42

> Thinking there should be a distribution independent way to figure out if we already have a network connection or not.

There is, but we need more than "is networking up"; rather we need to translate the existing configuration and merge that with whatever else may come from IMDS; in Oracle the iscsiroot has a permanent dhcp config on a specific interface, however IMDS can provide network config for additional interfaces, so we must merge them. The OCI datasource already does this but distros need to provide an initramfs network config parser to extract the network config generated in the initramfs to something cloud-init can understand.

[Robert Schweikert (rjschwei)]

Sorry for being dense, by the time we get to the point where we decide whether or not to bring up an ephemeral network we have long left the initrd and Since we are booting over iscsi the network is up and configured. Any configuration information we might need can be extracted from the network via "ip" commands. Those are distro independent thus a generic "translator" "live_config_to_net_cfg" would work everywhere. What am I missing?

[Ryan Harper (raharper)]

On Fri, Jan 17, 2020 at 15:15 Robert Schweikert <email address hidden>
wrote:

> Sorry for being dense, by the time we get to the point where we decide
> whether or not to bring up an ephemeral network we have long left the
> initrd and Since we are booting over iscsi the network is up and
> configured. Any configuration information we might need can be extracted
> from the network via "ip" commands. Those are distro independent thus a
> generic "translator" "live_config_to_net_cfg" would work everywhere.
> What am I missing?

The initrd supports more than just dhcp or static ip config and ip commands
won’t tell you which was used. There may be dns or other options, so it’s
best to parse the initramfs format which parses the kernel command line
anyhow to bring up networking in the initramfs.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1860164
>
> Title:
> cloud-init generates a traceback if a default route already exists
> during ephemeral network setup
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1860164/+subscriptions
>

[Robert Schweikert (rjschwei)]

Removing myself as assignee as I really have no idea right of the bat what we are after here and I will most likely not have the time to dig into all the gory details.

Here is the doc for what is supported w.r.t. configuration in dracut for these types of situation:

http://man7.org/linux/man-pages/man7/dracut.cmdline.7.html

and in detail the way the network would be configured:

ip={dhcp|on|any|dhcp6|auto6|either6}

           dhcp|on|any
               get ip from dhcp server from all interfaces. If root=dhcp,
               loop sequentially through all interfaces (eth0, eth1, ...)
               and use the first with a valid DHCP root-path.

           auto6
               IPv6 autoconfiguration

           dhcp6
               IPv6 DHCP

           either6
               if auto6 fails, then dhcp6

       ip=<interface>:{dhcp|on|any|dhcp6|auto6}[:[<mtu>][:<macaddr>]]
           This parameter can be specified multiple times.

           dhcp|on|any|dhcp6
               get ip from dhcp server on a specific interface

           auto6
               do IPv6 autoconfiguration

           <macaddr>
               optionally set <macaddr> on the <interface>. This cannot be
               used in conjunction with the ifname argument for the same
               <interface>.

       ip=<client-IP>:[<peer>]:<gateway-IP>:<netmask>:<client_hostname>:<interface>:{none|off|dhcp|on|any|dhcp6|auto6|ibft}[:[<mtu>][:<macaddr>]]
           explicit network configuration. If you want do define a IPv6
           address, put it in brackets (e.g. [2001:DB8::1]). This parameter
           can be specified multiple times. <peer> is optional and is the
           address of the remote endpoint for pointopoint interfaces and it
           may be followed by a slash and a decimal number, encoding the
           network prefix length.

           <macaddr>
               optionally set <macaddr> on the <interface>. This cannot be
               used in conjunction with the ifname argument for the same
               <interface>.

       ip=<client-IP>:[<peer>]:<gateway-IP>:<netmask>:<client_hostname>:<interface>:{none|off|dhcp|on|any|dhcp6|auto6|ibft}[:[<dns1>][:<dns2>]]
           explicit network configuration. If you want do define a IPv6
           address, put it in brackets (e.g. [2001:DB8::1]). This parameter
           can be specified multiple times. <peer> is optional and is the
           address of the remote endpoint for pointopoint interfaces and it
           may be followed by a slash and a decimal number, encoding the
           network prefix length.

ifname=<interface>:<MAC>
           Assign network device name <interface> (i.e. "bootnet") to the
           NIC with MAC <MAC>.

               Warning
               Do not use the default kernel naming scheme for the interface
               name, as it can conflict with the kernel names. So, don’t use
               "eth[0-9]+" for the interface name. Better name it "bootnet"
               or "bluesocket".

       rd.route=<net>/<netmask>:<gateway>[:<interface>]
           Add a static route with route options, which are separated by a
           colon. IPv6 addresses have to be put in b...

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3595