check_instance_id() error on reboots when using config-drive

[bugproxy]

Problem Description
=================================
When using a config-drive to provide meta-data to cloud-init on ubuntu (for Linux guest running in KVM for z Systems) we get a check_instance_id() error whenever we soft reboot after the (successful) initial boot.

The error shows:

[ 5.283203] cloud-init[1637]: Cloud-init v. 0.7.7 running 'init-local' at Sat, 23 Apr 2016 00:50:58 +0000. Up 5.25 seconds.
[ 5.283368] cloud-init[1637]: 2016-04-22 20:50:58,839 - util.py[WARNING]: failed of stage init-local
[ 5.286659] cloud-init[1637]: failed run of stage init-local
[ 5.286770] cloud-init[1637]: ------------------------------------------------------------
[ 5.286849] cloud-init[1637]: Traceback (most recent call last):
[ 5.286924] cloud-init[1637]: File "/usr/bin/cloud-init", line 520, in status_wrapper
[ 5.286998] cloud-init[1637]: ret = functor(name, args)
[ 5.287079] cloud-init[1637]: File "/usr/bin/cloud-init", line 250, in main_init
[ 5.287152] cloud-init[1637]: init.fetch(existing=existing)
[ 5.287225] cloud-init[1637]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 322, in fetch
[ 5.287298] cloud-init[1637]: return self._get_data_source(existing=existing)
[ 5.287371] cloud-init[1637]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 229, in _get_data_source
[ 5.287445] cloud-init[1637]: ds.check_instance_id(self.cfg)):
[ 5.287518] cloud-init[1637]: TypeError: check_instance_id() takes 1 positional argument but 2 were given
[ 5.287592] cloud-init[1637]: ------------------------------------------------------------
[FAILED] Failed to start Initial cloud-init job (pre-networking).

The failure of the init-local pre-networking does seem to lead to a boot up delay as cloud-init tries to search for networking outside of the already saved networking data.

Otherwise the error is purely cosmetic as later init modules find (or let existing IP configuration take over) and bring up the correct interfaces.

The original problem was found outside of openstack with stand-alone cloud-config iso images. But have been able to reproduce the problem within an openstack ICM environment.

Team has had some success getting around the problem by patching the check_instance_id function in /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py so that it accepted an extra argument, ex:

ubuntu@markvercd:~$ sudo cat check_instance_id.patch
--- /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py 2016-04-06 15:29:59.000000000 +0000
+++ /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py.new 2016-04-11 22:53:47.799867139 +0000
@@ -155,7 +155,7 @@

         return True

- def check_instance_id(self):
+ def check_instance_id(self,somecfg):
         # quickly (local check only) if self.instance_id is still valid
         return sources.instance_id_matches_system_uuid(self.get_instance_id())

ubuntu@markvercd:~$

---uname output---
Linux k6mpathcl.pokprv.stglabs.ibm.com 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:31:26 UTC 2016 s390x s390x s390x GNU/Linux

Machine Type = KVM guest on a z13 (2827-732) LPAR

Steps to Reproduce
=============================
 1) set up ubuntu guest image with cloud-init
2) pass in iso image with cloud-config data in cdrom device
3) boot up successfully with cloud-config data
4) attempt a soft reboot.

[Scott Moser (smoser)]

Thanks for the bug report and the fix.
Committed in trunk. I will try to get this uploaded to yakkety and to xenial today.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1215-0ubuntu1

---------------
cloud-init (0.7.7~bzr1215-0ubuntu1) yakkety; urgency=medium

  * debian/new-upstream-snapshot: minor change supporting revision
    passed in as an argument.
  * New upstream snapshot.
    - Paths: fix instance path if datasource's id has a '/'. (LP: #1575938)
    - Config Drive: fix check_instance_id signature. (LP: #1575055)
    - cloudstack: Only use DHCPv4 lease files as a datasource (LP: #1576273)

 -- Scott Moser <email address hidden> Fri, 29 Apr 2016 12:37:48 -0400

[Remy van Elst (raymii)]

Running OpenStack with ConfigDrive here and have this issue. Can confirm that the patches in '0.7.7~bzr1215-0ubuntu1' fix the error.

[Scott Moser (smoser)]

I'm pretty sure this is not fix-released in xenial.

[Juan J. Martínez (jjmartinez)]

I'm experiencing this on Xenial.

After the first reboot, cloud-init crashes with:

[ 7.987364] cloud-init[479]: Cloud-init v. 0.7.7 running 'init-local' at Wed, 22 Jun 2016 11:11:07 +0000. Up 7.89 seconds.
[ 7.994869] cloud-init[479]: 2016-06-22 11:11:07,161 - util.py[WARNING]: failed of stage init-local
[ 8.008421] cloud-init[479]: failed run of stage init-local
[ 8.016678] cloud-init[479]: ------------------------------------------------------------
[ 8.029215] cloud-init[479]: Traceback (most recent call last):
[ 8.037082] cloud-init[479]: File "/usr/bin/cloud-init", line 520, in status_wrapper
[ 8.049074] cloud-init[479]: ret = functor(name, args)
[ 8.060376] cloud-init[479]: File "/usr/bin/cloud-init", line 250, in main_init
[ 8.072385] cloud-init[479]: init.fetch(existing=existing)
[ 8.081004] cloud-init[479]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 322, in fetch[FAILED] Failed to start Initial cloud-init job (pre-networking).
See 'systemctl status cloud-init-local.service' for details.
[ 8.099419] cloud-init[479]: return self._get_data_source(existing=existing)

[Juan J. Martínez (jjmartinez)]

Sorry, one log line was missing:

cloud-init[479]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 229, in _get_data_source

[Juan J. Martínez (jjmartinez)]

And the last one:

[ 8.122969] cloud-init[479]: TypeError: check_instance_id() takes 1 positional argument but 2 were given

[Scott Moser (smoser)]

Juan,
its fixed in cloud-init trunk and in yakkety. It will be SRU'd shortly.

[Juan J. Martínez (jjmartinez)]

I know it's been fixed since April, I was expecting the fix to be already in the latest LTS.

Thanks Scott!

[Scott Moser (smoser)]

Hello,
An SRU upload of cloud-init for 16.04 that contains a fix for this bug has been made under bug 1595302. Please track that bug if you are interested.

[Scott Moser (smoser)]

fix is now released to xenial under bug 1595302. daily cloud-images with this newer version of cloud-init should appear in the next few days. Any image with a serial number *newer* than 20160707 should have cloud-init at 0.7.7~bzr1246-0ubuntu1~16.04.1 .

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-08-05 18:54 EDT-------
This seems to be working now.

Did the following:
- Created a base image that included the cloud-init material. Cloned it and provided the cloudconfig drive. Then did "sudo reboot" to soft reboot. Verified no more warnings at the "init-local" stage. Verified successful configuration from the cloudconfig drive.
- Ejected the cloudconfig drive. Did another soft reboot. Confirmed the boot messages still looked good.

Tested with the following levels:

ubuntu@ubuntu03:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
ubuntu@ubuntu03:~$
ubuntu@ubuntu03:~$ sudo apt-cache show cloud* | egrep "^Package|^Version"
Package: cloud-guest-utils
Version: 0.27-0ubuntu24
Package: cloud-initramfs-copymods
Version: 0.27ubuntu1
Package: cloud-initramfs-dyn-netconf
Version: 0.27ubuntu1
Package: cloud-init
Version: 0.7.7~bzr1256-0ubuntu1~16.04.1
ubuntu@ubuntu03:~$

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.7