seed ISO network changes fails to function

[Master James]

Ubuntu 19.04 with cloud-init 19

This is a local system while building an Ubuntu VM to eventually port to AWS, so internal systems.
[Eventually meta-data will be provided from the network as they do, but initially the seed.iso is the first layer to begin learning and developing for cloud-init.]

I tried many things for many days and could easily get the meta-data local-hostname to work.
This proves the system was correctly building and reading the cloud-init ISO mounting in VirtualBox as a seed.iso with files meta-data, user-data, and included network-config as the spec says should work.
Obviously the meta-data worked to change the local hostname. The 'user-data' file also worked as I will describe but network-config and the various methods of setting network info in meta-data, etc. failed to make any network changes to the booting system.

The first mystery I sold in this overly complected design in the end led to a hacky solution.
This is how the file /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg gets written to netplan's /etc/netplan/50-cloud-init.yaml so that 'netplan apply' can work it's magic.

It would seem the file in /etc/netplan/50-cloud-init.yaml would need to get written to.
In case that is to be overrideen by seed.iso I tried disabling as recommended 99-disable-network-config.cfg in hopes the seed.iso would take over instead but that now seems wrong attempt to understand what is needed. seeting dsmode to 'local' in tyhe meta-data also did nothing (again this is with multiple methods for setting networking version 2 in either meta-data or the network-config with mac matches, direct references to enp0s3 the network device name etc.

I learned the only and best cloud-init way to get it to actually write those changes is reflected in the hacky solution I now depend on to be able to set this via a seed.iso.

Over write the file 50-curtin-networking.cfg then run the cloud-init commands to clean the logs and restart the init-local service and ultimately apply the netplan networking changes.

This only works in the user-data file in the seed.iso mounted by VirtualBox during startup if it already has run cloud-init clean --logs so it does qualify as first run for these settings.

write_files:
  - path: /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg
    permissions: 0644
    owner: root
    content: |
      network:
          ethernets:
              enp0s3:
                  addresses: [192.168.1.11/24]
                  gateway4: 192.168.1.1
                  dhcp4: false
          version: 2

runcmd:
  - echo "Restarting cloud-init Local (networking)"
  - cloud-init clean --logs
  - cloud-init init --local
  - netplan apply

I would expect the seed.iso to over ride all other settings and become permanent and not need to be told to 'clean' so it should allow you to change it's IP and other settings at boot (regardless of first-boot), it is acceptable that the image has been issued a clean before shutdown prior to subsequent repeated scaling launches.

Note: it will be important to enable IPv4 and IPv6 addresses for each CPU on the Virtual device.

Anyway instead of going over the days of toil (path of discovery) I can assure many comprehensive attempts where tried before arrive at this only hacky solution.
It needs looking into. I will attempt to explain what was tried and report back to any suggestions of how this is suppose to be done if I misunderstood and so if that's not already documented there is at minimum a problem on that level.

[Dan Watkins (oddbloke)]

Hello!

Thanks for using cloud-init, and for filing this bug. It's not totally clear to me how I could reproduce the issue you're seeing. Could you give a clear list of reproduction steps, the actual behaviour you observe, the behaviour you would _expect_ to observe, and the tarball produced by `cloud-init collect-logs` on a failing instance, please?

Thanks!

Dan

[Dan Watkins (oddbloke)]

(Once you've provided the details, please set this back to New.)

[Master James (masterjames)]

Okay so I was looking into this some more and here is what I've learned about the problem.
So with an Ubuntu 19.04 VM (on VirtualBox via win10)
You have to run
cloud-init clean --logs
so the system will pick up the seed.iso on the next boot.
This makes sense because of security otherwise one could just change the passwords and gain access to any system. I would consider (am looking for) a config flag to let the system know booting from the seed.iso or nocloud-net etc. is a priority (or what order like boot-order in the bios of all computers), and not a concern in a given environment (so each boot a system can get a new IP sent through nocloud-net URLs from typically metadata_urls: [http://169.254.169.254:80] or the test seed.iso without calling cloud-init clean --logs at every shutdown first?

If I also delete /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg
with
network:
  ethernets:
    enp0s3:
      dhcp4: true
  version: 2

then the seed.iso network-config is used.
If you don't remove that file it will not.
This is a bug maybe know that the order of booting network is wrong and that it should do the /etc/cloud/cloud.cfg.d/ folder first and then override with changes from seed.iso's network-config file.

Note: I may have added the network: section unnecessarily to my network-config before this is what works.

#cloud-config
#vim:syntax=yaml

ethernets:
    enp0s3:
        addresses: [192.168.1.114/24]
        gateway4: 192.168.1.1
        nameservers:
            addresses: [127.0.0.1,192.168.1.1,8.8.8.8,8.8.4.4]
        dhcp4: false
version: 2

but only if you damage, disable, or remove the system file /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg

building the seed.iso is well documented. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-2-virtual-machine.html
Don't bother with the old network as we are all moving forward with the new way version 2.
https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html

Using meta-data local-hostname is a good way to confirm your seed.iso is working like so

local-hostname: myserver114

Anyway this discovery that the removal of the 50-curtin-networking.cfg is required for network-config on seed.iso to be recognized is going to bring clarity to this bug maybe it needs a title change?

Okay so if someone can please explain what I am missing that makes this all work really easily as expected then that is what should go into the documentation. Possible starting with...

If you plan to use a seed.iso you will need to run 'cloud-init clean' (prior to each boot) and delete your /etc/cloud/cloud.cfg.d/*networking.cfg files to get the seed.ISO's 'network-config' file to load/work.

...assuming it is not a bug of some kind that will allow seed.iso to work otherwise (aka as expected).

Thanks for listening and your help in making cloud-init functional for the community.

[Ryan Harper (raharper)]

Hi,

I think the missing bit of information is the instance-id. Cloud-init only performs certain operations at particular frequency; many things are per-instance. In your seed.iso, the meta-data file includes a key/value pair: instance-id: my-vm-1 ; something like that.

This value is used to keep track of whether cloud-init has configured this instance yet or not.
Cloud-init looks into /var/lib/cloud/instances/<instance-id> which is a directory which contains per-instance information about what cloud-init has (or has not yet) done.

Networking, for example, is typically written out only _once_ per-instance. In your test-cases, I suspect you've changed portions of the seed.iso, but not the instance-id value; and as you found out; the only way to re-test is to use cloud-init clean --logs (which will remove the instance directory among other things).

[Master James (masterjames)]

Okay well I had seen and understood that about the instance-id in meta-data. Let's try...

I have a meta-data file with just this.

#cloud-config
#vim:syntax=yaml

instance-id: myserver-id111
local-hostname: myserver111

If I change the ISO meta-data file's instance-id and local-hostname to *002 (including the network-config IP that works prior's IP to be different) and simply reboot the VM via
shutdown -r now
it comes back but does not change the hostname or IP or anything unless you run 'cloud-init clean'

network-config looks like this for ubuntu 19.04...
#cloud-config
#vim:syntax=yaml

ethernets:
    enp0s3:
        addresses: [192.168.3.111/24]
        gateway4: 192.168.3.1
        dhcp4: false
version: 2

If I disconnect the seed.ISO and reboot it retains last settings fine.
If I 'cloud-init clean' and 'reboot' it will revert to dhcp for addresses but keep the hostname, [which I suppose is expected behavior, but I just thought I'd should add that fact].

Even though /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg
has been deleted, which would have this...
network:
  ethernets:
    enp0s3:
      dhcp4: true
  version: 2

technically with the 50-curtin-networking.cfg networking setting file destroyed maybe it should keep what it had but I guess dhcp is default.

I think maybe the cloud-init network-config should be checked if it was last first booted from it for different IP settings per boot as I imagine it should from a meta-data_url or 'net' data-source.

I will investigate eventually how cloud-init might work differently in this regard when a https://*.254 URL is present but it seems like every boot you'd need to reset the cloud-init to get different IPs at boot without just relying on a regular DHCP being hosted by default (or otherwise) on a given VPC (Virtual Private Cloud) network.

[Master James (masterjames)]

I am looking at https://cloudinit.readthedocs.io/en/latest/topics/boot.html#network
and slightly confused where I would find the network IP configuration to run per boot from network-config via seed.iso or URLs, since it says the "Networking" stage requires the networks to all be up already?
I think it's named wrong Networking should be labeled "Disk Mounting & User Data" or something?

Anyway I am thinking now where is it configured to run networking once and move it to per-boot status? Which it seems maybe is the bug or needs to be a config flag or something?

[Dan Watkins (oddbloke)]

Hi!

Could you provide the `cloud-init collect-logs` output from the instance after you've changed the metadata, please?

The Network stage is named because it _requires_ network, not because it configures network. The docs are pretty clear about this: "After local stage and configured networking is up." "This stage requires all configured networking to be online". (They're also, I think, pretty clear that the Local stage's purpose is to "apply networking configuration to the system".)

Thanks!

Dan

[Master James (masterjames)]

Unfortunately I can only provide limited sanitized logs because of NDAs & Security reasons.
If there is a particular portion of interest maybe I can redact something.

I have basically resolved to use DHCP in cloud and then assign-private-ip-addresses and associate-address with elastic public IPs per CPU etc.

I don't think I am going to pursue building and analyzing an internal cloud-init server to see if it respects it like DHCP would to give a new IP on the fly when restarting etc. without needed to do a 'cloud-init clean' in a per-boot way.

Maybe that comment brings clarity to my point of how it seems to me seed.iso network-config file should work on reboots. I thought it's possible I am simply missing something simple? It seems to me a flag or setting to tell the system to observe changes to the netowrk-config file on any seed.iso (aka -volid cidata) on a per-boot makes the most sense to me.

If anyone can't point to how to rig cloud-init to do so or bring clarity to me in these regards that would be appreciated, thanks.

[Dan Watkins (oddbloke)]

Hi,

I'm afraid if you can't provide us with the debugging information we need, we can't really help much more. Are you able to reproduce the issue in a non-NDA environment, and provide information from there?

Thanks,

Dan

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3407