cloud-init creates /etc/network/interfaces.d/50-cloud-init.cfg even if /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg present

[Paul Graydon]

Package: cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.3

The file /etc/network/interfaces.d/50-cloud-init.cfg states that:

# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}

On a system with this file created:

# cat /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}

cloud-init still ends up creating the 50-cloud-init.cfg file. opensnoop shows that cloud-init is definitely reading that 99-disable-network-config.cfg file:

# opensnoop /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
Tracing open()s for filenames containing "/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg". Ctrl-C to end.
COMM PID FD FILE
cloud init 0x3 /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
....

I've been trying to follow the code logic, and I'm not seeing any check to see if the network configuration is set to disabled or not.

[Scott Moser (smoser)]

cloudinit/stages.py uses 'net.is_disabled_cfg()'.

I just verified on a fresh lxc container.

 * boot system
 * lxc exec x1
   % echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
   % rm /etc/network/interfaces.d/50-cloud-init.cfg
   % reboot
 * lxc exec x1
   % cat /etc/network/interfaces.d/50-cloud-init.cfg
   cat: /etc/network/interfaces.d/50-cloud-init.cfg: No such file or directory
   % dpkg-query --show cloud-init
   cloud-init 0.7.8-1-g3705bb5-0ubuntu1~16.04.3

   % grep "INFO.*disabled" /var/log/cloud-init.log
   Nov 10 02:07:59 xt1 [CLOUDINIT] stages.py[INFO]: network config is disabled by system_cfg
   Nov 10 02:08:00 xt1 [CLOUDINIT] stages.py[INFO]: network config is disabled by system_cfg

So it seems not completely broken there.

[Paul Graydon (pgraydon-oracle)]

Clean instance, ran apt upgrade, and apt install cloud-init. I didn't replace the cloud.cfg file with one of our format as I tried to keep changes as minimal as possible, and the supplied one seems to work sufficiently.

$ echo "network: {config: disabled}" | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}

$ sudo rm /etc/network/interfaces/50-cloud-init.cfg
rm: cannot remove '/etc/network/interfaces/50-cloud-init.cfg': Not a directory

$ sudo reboot
packet_write_wait: Connection to UNKNOWN: Broken pipe

After reboot:

$ cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback

# control-manual ens2f0
iface ens2f0 inet dhcp
    broadcast 10.0.0.255
    dns-nameservers 169.254.169.254
    gateway 10.0.0.1
    netmask 255.255.255.0

$ grep "INFO.*disabled" /var/log/cloud-init.log
$

I've attached a copy of the cloud-init.log

[Paul Graydon (pgraydon-oracle)]

Trying another time. Again, clean instance. This time using a very minimal cloud.cfg:

$ cat /etc/cloud/cloud.cfg
users:
   - default

datasource_list: ['OpenStack']
datasource:
   OpenStack:
     metadata_urls: ['http://169.254.169.254']
     timeout: 10
     max_wait: 20

cloud_init_modules:
 - users-groups

system_info:
   distro: ubuntu
   default_user:
    name: opc
    lock_passwd: true
    gecos: Oracle Public Cloud User
    groups: [wheel, adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
   paths:
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
      upstart_dir: /etc/init/
   ssh_svcname: ssh

$ echo "network: {config: disabled}" | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
network: {config: disabled}
$ sudo rm /etc/network/interfaces/50-cloud-init.cfg
rm: cannot remove '/etc/network/interfaces/50-cloud-init.cfg': Not a directory

And again after a reboot:

$ cat /etc/network/interfaces.d/50-cloud-init.cfg
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
auto lo
iface lo inet loopback

# control-manual ens2f0
iface ens2f0 inet dhcp
    broadcast 10.0.0.255
    dns-nameservers 169.254.169.254
    gateway 10.0.0.1
    netmask 255.255.255.0

$ grep "INFO.*disabled" /var/log/cloud-init.log
$

I've attached a cloud-init log from this run as well. Hopefully should be more succinct.

[Scott Moser (smoser)]

ok, so
 Nov 10 17:17:26 ubuntu [CLOUDINIT] stages.py[INFO]: Applying network configuration from cmdline bringup=False: {'version': 1, 'config': [{'subnets': [{'gateway': '10.0.0.1', 'broadcast': '10.0.0.255', 'control': 'manual', 'type': 'dhcp', 'dns_nameservers': ['169.254.169.254'], 'netmask': '255.255.255.0'}], 'name': 'ens2f0', 'type': 'physical', 'mac_address': '90:e2:ba:cd:23:f8'}]}

Somewhat surprised / disappointed that cloud-init did not log your kernel command line, but can you put that here?

It apparently writes network config if it uses the kernel comand line as the source of network info.

I'm curious, what datasource do you have available to cloud-init ?

[Paul Graydon (pgraydon-oracle)]

Kernel command line:

$ cat /proc/cmdline
BOOT_IMAGE=(http)/kernel initrd=initrd root=/dev/sda3 ro netroot=iscsi:@169.254.0.2::3260::iqn.2015-02.oracle.boot:uefi crashkernel=auto ip=dhcp iscsi_initiator=iqn.2015-10.oracle:2.1g1538-gb000393 LANG=en_US.UTF-8 console=ttyS0,9600 console=tty0

The datasource is a yaml file we generate on instance launch that is made accessible to the host over http. At the moment it contains very little information:

$ curl http://169.254.169.254/openstack/latest/meta_data.json
{
  "uuid" : "ocid1.instance.oc1.phx.abyhqljszmao4waexfg6gxtu73phfgeclyyscoi6r3evhsaba24uupp3hu5q",
  "name" : "nvmetest",
  "availability_zone" : "phx-ad-2",
  "hostname" : "nvmetest",
  "public_keys" : {
    "0" : "<<redacted>>"
  },
  "meta" : { }
}

It's also available via a few different URLs, e.g. http://169.254.169.254/openstack/2013-10-17/meta_data.json will return the same content.

[Scott Moser (smoser)]

The way this is working in cloud-init seems to me to make some sense.
Basically, there are 3 places cloud-init can get its network configuration. In order of increasing precedence:
 1. datasource (only works for local datasources currently)
 2. system configuration
 3. kernel command line.
    a.) ip= argument: files written by the initramfs networking system (/run/net-eth0.conf) for example.

    b.) network-config= argument. This is yaml formated text blob.

So, if datasource (1) provides network config, but the system says not to pay attention to it (2), then cloud-init will not write network config. Similarly if system says it is disabled, but the kernel command line enables it, then cloud-init will write networking config.

You're kind of hitting a weird situation where the kernel cmdline has 3a, but you dont want cloud-init to render system based on that. (The reason you dont is a bug somewhere, but I'm not sure, as this does not happen in many iscsi root scenarios that we have and test).

One solution I think should work for you is if you pass on the kernel command line:
 network-config=eyJuZXR3b3JrIjp7ImNvbmZpZyI6ImRpc2FibGVkIn19fQ==

This is specifying the config to cloud-init in base64 encoded network config format (described at http://curtin.readthedocs.io/en/latest/topics/networking.html).

So what you have there is:
$ echo eyJuZXR3b3JrIjp7ImNvbmZpZyI6ImRpc2FibGVkIn19fQ== | base64 --decode ; echo
{"network":{"config":"disabled"}}}

Last thing... I see 'netroot=' on you command line.
I'm guessing that means you've generated an initramfs with dracut?
I do not know how well that would be supported in ubuntu. My experience with iscsi root is with initramfs-tools which would take a different form of kernel command line.

In 16.04 and later you should be able to use root=iscsi:<rfc4173>
 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=804162
I think that is also supported in dracut if you're looking for something supported by both.

[Paul Graydon (pgraydon-oracle)]

We can tackle the other aspects of this on other tickets, but it does seem like we have a good understanding of what is happening here. I want to make sure we don't end up derailing slightly here from the "bug" as it were (though it's not so much a bug, it appears)

Just operating from the principle of least-surprise, if cloud-init tells you in a file "create this file like so, and it'll stop this one being generated", I would expect that to be the case.

On the other hand adding in the conditionals to the comment seems like maybe it would be overloading the comments in the file? I like to keep such comments about as short and sweet as possible.

Would it be more realistic to get the behaviour documented and just point to the documents (but then I guess what about versioning?)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jason Mills (virtualjmills)]

Additional nuance:

On a netplan enabled system also using cloud-init (+ curtin), the same assertion is made regarding how to disable network functionality tied to cloud-init is mentioned in /etc/netplan/50-cloud-init.yaml ("… disable via stanza in /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg …") and /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg files.

… however …

If /etc/netplan/50-cloud-init.yaml is not deleted, that stanza is ignored.

Ubuntu 18.04.0 Server (via Bionic Live-Server 2018.03.19 daily)

No network related arguments included in kernel command line.

Initial install configuration was first-ethernet dhcp IPv4.

Neither ifupdown nor ifupdown2 not installed.

At the end of the installation phase + first reboot, both /etc/netplan/50-cloud-init.yaml and /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg contain the minimal definition (first-ethernet dhcp IPv4).