ConfigDrive: cloud-init fails to configure network from network_data.json

[Mathieu Gagné]

When running Ubuntu 16.04 on OpenStack, cloud-init fails to properly configure the network from network_data.json found in ConfigDrive.

When instance boots, network is configured fine until next reboot where it falls back to dhcp.

The /etc/network/interfaces.d/50-cloud-init.cfg file has the following content when instance is initially booted, this could explain why dhcp is used on second boot:

    auto lo
    iface lo inet loopback

    auto eth0
    iface eth0 inet dhcp

When debugging, if this line in stages.py [1] is commented, we can see that cloud-init initially copy the /etc/network/interfaces file found in the configdrive (the network template injected by Nova) and isn't using the network config found in network_data.json. But later it falls back to "dhcp" and rewrites yet again the network config.

I also found that within self._find_networking_config(), it looks like no datasource is found at this point. This could be because cloud-init is still in "local" dsmode and then refuses to use the network config found in the ConfigDrive. (triggering the "dhcp" fallback logic)

Manually forcing "net" dsmode makes cloud-init configure /etc/network/interfaces.d/50-cloud-init.cfg properly with network config found in the ConfigDrive. However no gateway is configured and so, instance doesn't respond to ping or SSH.

At that point, I'm not sure what's going on and how I can debug further.

Notes:
* The image used for testing uses "net.ifnames=0". Removing this config makes things much worst. (no ping at all on first boot)
* Logs, configs and configdrive can be found attached to this bug report.

[1] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/stages.py#L604

[Michal Adamczyk (vanditboy)]

Hi,

My cloud platform is OpenStack Mitaka on RDO (CentOS 7.2) with XEN hypervisor and config_drive option.

On my guest system is CentOS 6.7.

Cloud-init agent can pickup meta_data.json (as I can login via ssh when bring up the network manually) but it cannot setup network:

Starting cloud-init: Cloud-init v. 0.7.5 running 'init-local' at Wed, 18 May 2016 08:19:18 +0000. Up 22.67 seconds.
Kdump is not supported on this kernel [FAILED]
Starting cloud-init: Cloud-init v. 0.7.5 running 'init' at Wed, 18 May 2016 08:19:18 +0000. Up 23.39 seconds.
ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0 | False | . | . | fa:16:3e:7e:8f:f5 |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2016-05-18 08:19:19,566 - util.py[WARNING]: Failed forking and calling callback NoneType
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:config' at Wed, 18 May 2016 08:19:20 +0000. Up 24.60 seconds.
2016-05-18 08:19:20,623 - util.py[WARNING]: Failed to run debconf-set-selections for grub-dpkg
2016-05-18 08:19:20,651 - util.py[WARNING]: Running apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_apt_configure.pyc'>) failed
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:final' at Wed, 18 May 2016 08:19:20 +0000. Up 25.41 seconds.
ci-info: ++++++++++++++++Authorized keys from /home/centos/.ssh/authorized_keys for user centos+++++++++++++++++
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ci-info: | ssh-rsa | f3:f2:08:9f:cd:fd:1b:67:4c:9b:57:b7:5f:c6:54:e5 | - | - |
ci-info: | ssh-rsa | 75:3e:12:1e:09:51:1e:b9:6d:79:fd:4d:84:31:5d:51 | - | <email address hidden> |
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 5a:70:be:fb:bf:67:1b:ce:12:a6:7e:ba:ef:04:4f:ef /etc/ssh/ssh_host_dsa_key.pub (DSA)
ec2: 2048 50:22:cd:6d:1c:56:eb:10:42:54:97:b6:7a:81:41:f1 /etc/ssh/ssh_host_key.pub (RSA1)
ec2: 2048 15:55:5d:e6:91:af:46:3d:3a:bb:6f:f6:0a:e6:03:6f /etc/ssh/ssh_host_rsa_key.pub (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
2048 35 29586527545742027782438580921178232663437101143928703426912025054688176...

[Dan Watkins (oddbloke)]

Hi Michal,

Consumption of network_data.json was introduced in trunk at some point before revision 1212; you'll need to update to use a more recent version of cloud-init to get it working.

Thanks,

Dan

[Scott Moser (smoser)]

Hi, anyone listening I'd like test of my ppa at https://launchpad.net/~smoser/+archive/ubuntu/cloud-init-dev .

You should be able to test by:
* clean xenial (or yakkety or wily) instance
* you can add a local user with a password interactively here to allow you back in if it fails to get neworking.
* sudo sh -c 'apt-add-repository -y ppa:smoser/cloud-init-dev && sudo apt-get update && sudo apt-get install cloud-init -qy'
* sudo reboot # [check that it works fine]
  look to see that there are no 'WARN' values in /var/log/cloud-init.log
* sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
* sudo reboot
  look to see WARN values

If this does not work for you, please attach the config drive that was used for me to look at (you can just 'dd if=/dev/sr0 | gzip -c > my.iso.gz'

And if you see any WARN in logs, please attach them.
thank you.

[Scott Moser (smoser)]

This was fixed in yakkety at 0.7.7~bzr1227-0ubuntu1 . It will be sru'd sometime soon.

[Scott Moser (smoser)]

This was fixed in trunk at revno 1225

[Scott Moser (smoser)]

Hi,
This bug was fixed in cloud-init at revision 1225 and was sru'd to xenial under bug 1595302, but unfortunately not even marked in that bug.

So anything newer than 0.7.7~bzr1245-0ubuntu1~16.04.1 in xenial should have the fix.

[long (long-cheung)]

Hello scott,

i using cloud-init Nocloud have same problem,
Hostname and password can setup,
but network can not,i using same config in ubuntu 14.04 is work,
but using 16.04 can not setup the network.Can you help me to check?

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.7

[long (long-cheung)]

Dear scott,

Thank for your reply,
But i already using 0.7.7~bzr1256-0ubuntu~16.04.1
Maybe i have something wrong,But i already check 1week,
Can not fix,....Can you please help to try?

[long (long-cheung)]

Dear scott,

Follow is my meta-data and user-date,i using nocloud

instance-id: 1470899540
local-hostname: soscared
network-interfaces: |
    auto eth0
    iface eth0 inet static
    address 103.225.10.12
    netmask 255.255.255.0
    gateway 103.225.10.1
    dns-servers 8.8.8.8

#cloud-config
ssh_pwauth: True
chpasswd:
  list: |
     root:sosadc123456
  expire: False

packages:
  - vim
  - qemu-guest-agent
  - zsh

package_upgrade: true

runcmd:
  - reboot

# vim: syntax=yaml list

Then i run the cloud-init,Hostname and password can setup,But network is fail.

i check in obj.pkl ip format is wrong,
Is some in put setting wrong or cloud-init gen the format is wrong?

[Scott Moser (smoser)]

Hi Long,
could you please open a different bug? and link to it from here?

This bug specifically handled 'config-drive' (the openstack disk format rather than 'NoCloud').

I have reproduced your issue.

[Scott Moser (smoser)]

Hi Long,
I went ahead and opened a bug at
 https://bugs.launchpad.net/cloud-init/+bug/1612231

[long (long-cheung)]

Dear Scott,

I update some comment in bugs,

Can you please help,

Thank for your help.

Long

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Scott Moser
Sent: Thursday, August 11, 2016 8:43 PM
To: Long Cheung
Subject: [Bug 1577982] Re: ConfigDrive: cloud-init fails to configure network from network_data.json

Hi Long,
I went ahead and opened a bug at
 https://bugs.launchpad.net/cloud-init/+bug/1612231

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1577982

Title:
  ConfigDrive: cloud-init fails to configure network from
  network_data.json

Status in cloud-init:
  Fix Released
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Fix Released

Bug description:
  When running Ubuntu 16.04 on OpenStack, cloud-init fails to properly
  configure the network from network_data.json found in ConfigDrive.

  When instance boots, network is configured fine until next reboot
  where it falls back to dhcp.

  The /etc/network/interfaces.d/50-cloud-init.cfg file has the following
  content when instance is initially booted, this could explain why dhcp
  is used on second boot:

      auto lo
      iface lo inet loopback

      auto eth0
      iface eth0 inet dhcp

  When debugging, if this line in stages.py [1] is commented, we can see
  that cloud-init initially copy the /etc/network/interfaces file found
  in the configdrive (the network template injected by Nova) and isn't
  using the network config found in network_data.json. But later it
  falls back to "dhcp" and rewrites yet again the network config.

  I also found that within self._find_networking_config(), it looks like
  no datasource is found at this point. This could be because cloud-init
  is still in "local" dsmode and then refuses to use the network config
  found in the ConfigDrive. (triggering the "dhcp" fallback logic)

  Manually forcing "net" dsmode makes cloud-init configure
  /etc/network/interfaces.d/50-cloud-init.cfg properly with network
  config found in the ConfigDrive. However no gateway is configured and
  so, instance doesn't respond to ping or SSH.

  At that point, I'm not sure what's going on and how I can debug
  further.

  Notes:
  * The image used for testing uses "net.ifnames=0". Removing this config makes things much worst. (no ping at all on first boot)
  * Logs, configs and configdrive can be found attached to this bug report.

  [1] http://bazaar.launchpad.net/~cloud-init-dev/cloud-
  init/trunk/view/head:/cloudinit/stages.py#L604

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1577982/+subscriptions

[Peter (pry)]

Hello, Scott!

Looks like the problem that Long stated in the post #8 persists and it wasn't fixed in 0.7.7.

I perform cloud image configuration using the 'network-interfaces' key in meta-data file. First boot goes smooth, I detach the iso file and reboot and the VM falls back to DHCP. I already have net.ifnames=0 in my kernel parameters. I'm using version 0.7.8 and it doesn't seem to be fixed there.

P.S. You've opened the bug in #13, but it looks like it's a different bug so i decided to write here. I'll open a separate bug if it is needed.

Some info:

$ cloud-init -v
cloud-init 0.7.8

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.8.0-27-generic root=UUID=abe2efde-1980-4e3f-bb55-b7bf90872ef9 ro console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0 ds=nocloud

Running Ubuntu 16.10 x86_64

[Deepa (dpaclt)]

I do have the same issue .

#cloud-init -v
cloud-init 0.7.9

Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

[Marc Siegrist (dev-marc)]

This still affects me with cloud-init version 0.7.9 and Ubuntu 16.04.03 Lts.

[David Britton (dpb)]

Hi Marc, Deepa, Peter --

It's probably best to open a new bug on this. If you wouldn't mind doing so with a clear reproduction in the form of "what happened, what I expected, observations made" that would be very helpful!

Thanks.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2666