check_instance_id() error on reboots when using config-drive

Bug #1575055 reported by bugproxy on 2016-04-26
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Medium
Unassigned
cloud-init
High
Scott Moser
cloud-init (Ubuntu)
Medium
Scott Moser
Xenial
Medium
Unassigned

Bug Description

Problem Description
=================================
When using a config-drive to provide meta-data to cloud-init on ubuntu (for Linux guest running in KVM for z Systems) we get a check_instance_id() error whenever we soft reboot after the (successful) initial boot.

The error shows:

[ 5.283203] cloud-init[1637]: Cloud-init v. 0.7.7 running 'init-local' at Sat, 23 Apr 2016 00:50:58 +0000. Up 5.25 seconds.
[ 5.283368] cloud-init[1637]: 2016-04-22 20:50:58,839 - util.py[WARNING]: failed of stage init-local
[ 5.286659] cloud-init[1637]: failed run of stage init-local
[ 5.286770] cloud-init[1637]: ------------------------------------------------------------
[ 5.286849] cloud-init[1637]: Traceback (most recent call last):
[ 5.286924] cloud-init[1637]: File "/usr/bin/cloud-init", line 520, in status_wrapper
[ 5.286998] cloud-init[1637]: ret = functor(name, args)
[ 5.287079] cloud-init[1637]: File "/usr/bin/cloud-init", line 250, in main_init
[ 5.287152] cloud-init[1637]: init.fetch(existing=existing)
[ 5.287225] cloud-init[1637]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 322, in fetch
[ 5.287298] cloud-init[1637]: return self._get_data_source(existing=existing)
[ 5.287371] cloud-init[1637]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 229, in _get_data_source
[ 5.287445] cloud-init[1637]: ds.check_instance_id(self.cfg)):
[ 5.287518] cloud-init[1637]: TypeError: check_instance_id() takes 1 positional argument but 2 were given
[ 5.287592] cloud-init[1637]: ------------------------------------------------------------
[FAILED] Failed to start Initial cloud-init job (pre-networking).

The failure of the init-local pre-networking does seem to lead to a boot up delay as cloud-init tries to search for networking outside of the already saved networking data.

Otherwise the error is purely cosmetic as later init modules find (or let existing IP configuration take over) and bring up the correct interfaces.

The original problem was found outside of openstack with stand-alone cloud-config iso images. But have been able to reproduce the problem within an openstack ICM environment.

Team has had some success getting around the problem by patching the check_instance_id function in /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py so that it accepted an extra argument, ex:

ubuntu@markvercd:~$ sudo cat check_instance_id.patch
--- /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py 2016-04-06 15:29:59.000000000 +0000
+++ /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceConfigDrive.py.new 2016-04-11 22:53:47.799867139 +0000
@@ -155,7 +155,7 @@

         return True

- def check_instance_id(self):
+ def check_instance_id(self,somecfg):
         # quickly (local check only) if self.instance_id is still valid
         return sources.instance_id_matches_system_uuid(self.get_instance_id())

ubuntu@markvercd:~$

---uname output---
Linux k6mpathcl.pokprv.stglabs.ibm.com 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:31:26 UTC 2016 s390x s390x s390x GNU/Linux

Machine Type = KVM guest on a z13 (2827-732) LPAR

Steps to Reproduce
=============================
 1) set up ubuntu guest image with cloud-init
2) pass in iso image with cloud-config data in cdrom device
3) boot up successfully with cloud-config data
4) attempt a soft reboot.

Related branches

bugproxy (bugproxy) on 2016-04-26
tags: added: architecture-s39064 bugnameltc-140675 severity-medium targetmilestone-inin1610
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
bugproxy (bugproxy) on 2016-04-26
tags: added: targetmilestone-inin16041
removed: targetmilestone-inin1610
affects: ubuntu → cloud-init (Ubuntu)
Changed in cloud-init (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → Scott Moser (smoser)
Scott Moser (smoser) on 2016-04-26
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser) wrote :

Thanks for the bug report and the fix.
Committed in trunk. I will try to get this uploaded to yakkety and to xenial today.

Changed in cloud-init:
assignee: nobody → Scott Moser (smoser)
importance: Undecided → High
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.7~bzr1215-0ubuntu1

---------------
cloud-init (0.7.7~bzr1215-0ubuntu1) yakkety; urgency=medium

  * debian/new-upstream-snapshot: minor change supporting revision
    passed in as an argument.
  * New upstream snapshot.
    - Paths: fix instance path if datasource's id has a '/'. (LP: #1575938)
    - Config Drive: fix check_instance_id signature. (LP: #1575055)
    - cloudstack: Only use DHCPv4 lease files as a datasource (LP: #1576273)

 -- Scott Moser <email address hidden> Fri, 29 Apr 2016 12:37:48 -0400

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Remy van Elst (raymii) wrote :

Running OpenStack with ConfigDrive here and have this issue. Can confirm that the patches in '0.7.7~bzr1215-0ubuntu1' fix the error.

Mathew Hodson (mhodson) on 2016-05-12
Changed in cloud-init (Ubuntu Xenial):
milestone: none → ubuntu-16.04.1
airah (jhonajamail) on 2016-06-03
Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Released
Scott Moser (smoser) wrote :

I'm pretty sure this is not fix-released in xenial.

Changed in cloud-init (Ubuntu Xenial):
status: Fix Released → Confirmed
Juan J. Martínez (jjmartinez) wrote :

I'm experiencing this on Xenial.

After the first reboot, cloud-init crashes with:

[ 7.987364] cloud-init[479]: Cloud-init v. 0.7.7 running 'init-local' at Wed, 22 Jun 2016 11:11:07 +0000. Up 7.89 seconds.
[ 7.994869] cloud-init[479]: 2016-06-22 11:11:07,161 - util.py[WARNING]: failed of stage init-local
[ 8.008421] cloud-init[479]: failed run of stage init-local
[ 8.016678] cloud-init[479]: ------------------------------------------------------------
[ 8.029215] cloud-init[479]: Traceback (most recent call last):
[ 8.037082] cloud-init[479]: File "/usr/bin/cloud-init", line 520, in status_wrapper
[ 8.049074] cloud-init[479]: ret = functor(name, args)
[ 8.060376] cloud-init[479]: File "/usr/bin/cloud-init", line 250, in main_init
[ 8.072385] cloud-init[479]: init.fetch(existing=existing)
[ 8.081004] cloud-init[479]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 322, in fetch[FAILED] Failed to start Initial cloud-init job (pre-networking).
See 'systemctl status cloud-init-local.service' for details.
[ 8.099419] cloud-init[479]: return self._get_data_source(existing=existing)

Juan J. Martínez (jjmartinez) wrote :

Sorry, one log line was missing:

cloud-init[479]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 229, in _get_data_source

Juan J. Martínez (jjmartinez) wrote :

And the last one:

[ 8.122969] cloud-init[479]: TypeError: check_instance_id() takes 1 positional argument but 2 were given

Scott Moser (smoser) wrote :

Juan,
its fixed in cloud-init trunk and in yakkety. It will be SRU'd shortly.

Juan J. Martínez (jjmartinez) wrote :

I know it's been fixed since April, I was expecting the fix to be already in the latest LTS.

Thanks Scott!

Frank Heimes (fheimes) on 2016-06-23
Changed in ubuntu-z-systems:
status: New → Fix Committed
importance: Undecided → Medium
Scott Moser (smoser) wrote :

Hello,
An SRU upload of cloud-init for 16.04 that contains a fix for this bug has been made under bug 1595302. Please track that bug if you are interested.

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → In Progress
Changed in cloud-init (Ubuntu Xenial):
status: In Progress → Fix Committed
Scott Moser (smoser) wrote :

fix is now released to xenial under bug 1595302. daily cloud-images with this newer version of cloud-init should appear in the next few days. Any image with a serial number *newer* than 20160707 should have cloud-init at 0.7.7~bzr1246-0ubuntu1~16.04.1 .

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released
Frank Heimes (fheimes) on 2016-07-11
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released

------- Comment From <email address hidden> 2016-08-05 18:54 EDT-------
This seems to be working now.

Did the following:
- Created a base image that included the cloud-init material. Cloned it and provided the cloudconfig drive. Then did "sudo reboot" to soft reboot. Verified no more warnings at the "init-local" stage. Verified successful configuration from the cloudconfig drive.
- Ejected the cloudconfig drive. Did another soft reboot. Confirmed the boot messages still looked good.

Tested with the following levels:

ubuntu@ubuntu03:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
ubuntu@ubuntu03:~$
ubuntu@ubuntu03:~$ sudo apt-cache show cloud* | egrep "^Package|^Version"
Package: cloud-guest-utils
Version: 0.27-0ubuntu24
Package: cloud-initramfs-copymods
Version: 0.27ubuntu1
Package: cloud-initramfs-dyn-netconf
Version: 0.27ubuntu1
Package: cloud-init
Version: 0.7.7~bzr1256-0ubuntu1~16.04.1
ubuntu@ubuntu03:~$

Scott Moser (smoser) wrote :

This is fixed in cloud-init 0.7.7

Changed in cloud-init:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers