Commissioning fails when competing cloud metadata resides on disk

[mahmoh]

=== Begin SRU Information ===
[Impact]
The issue originally reported was that when MAAS attempted to enlist a
system by booting it with a remote iscsi disk with intent to have cloud-init
utilize the MAAS metadata service, cloud-init found some metadata from a
previous use of the system on the local disk. cloud-init then went on
to use that data and did not respond to maas.

The impact in this case was that cloud-init failed to enlist. The same problem
could occur in any other case where there was data on the local disk that
provided a datasource for cloud-init.

The fix provided was for the scenario provided was for MAAS to provide
configuration on the maas provided kernel command line that tells cloud-init
it should read only attempt to use the MAAS datasource.

Specifically, as mentioned in comment 7, this looked like:
   root=iscsi:.... cc:{'datasource_list': ['MAAS']}end_cc \
       cloud-config-url=http://maas/path/to/config ...

cloud-init then reads that information on boot and it overrides the settings
found inside the iscsi root device.

[Test Case]
A test case lives in unit tests now that ensures kernel config overrides
system config.

To further test this we could
a.) cause this situation by
  1.) installing a node in maas
  2.) putting config drive or nocloud data onto one of the partitions
  3.) returning the system to maas
  4.) attempt re-deploy.

b.) use a cloud image, kernel and initramfs and web server
  1.) download image, update cloud-init to -proposed.
  2.) set up a web service to serve files like MAAS described at
      https://maas.ubuntu.com/docs/development/metadata.html
  3.) boot image with kernel command line including the cc: and cloud-config-url referencing that web service.
  4.) have provided a config drive or nocloud seed disk to the vm.

The 'b' test above is easier to reproduce in that it does not rely on MAAS.

[Regression Potential]
Regression potential is low, in that this feature worked for some time
in previous releases. A bad reading of the code made me (smoser) change
the code intending to fix the problem, but in fact regressed it. So this
change is actually reverting a previous change in behavior.

This was first broken in 16.04 in 0.7.7~bzr1245-0ubuntu1~16.04.1 .

[Other Info]
The upstream commit that fixed this behavior (including the added tests)
is 0b0f254a [1]

--
[1] https://git.launchpad.net/cloud-init/commit/?id=0b0f254a6935a1b1fff128fa177152dd519e1a3d

=== End SRU Information ===

A customer reused hardware that had previously deployed a RHEL Overcloud-controller which places metadata on the disk as a legitimate source, that cloud-init looks at by default. When the newly enlisted node appeared it had the name of "overcloud-controller-0" vs. maas-enlist, pulled from the disk metadata which had overridden MAAS' metadata. Commissioning continually failed on all of the nodes until the disk metadata was manually removed (KVM boot Ubuntu ISO, rm -f data or dd zeros to disk).

[mahmoh (mahmoh)]

MAAS version 2.0b3 & 2.0b4.

[Blake Rouse (blake-rouse)]

Scott,

What can MAAS do to tell cloud-init that its should use the MAAS datasource first and always in the enlistment and commissioning environment. We don't want cloud-init to look at the disk first in the case of MAAS.

[Andres Rodriguez (andreserl)]

Hold on a second, why the machine which is "commissioning" try to access the local data source ??

This doesn't make sense provided that commissioning runs on memory, and mounts a rootfs, hence, it would never see the cloud-init data what's on the disk.

[mahmoh (mahmoh)]

The cloud-init data is on the second disk, when is cloud-init run? During enlistment and commissioning?

It continually failed in the process until we booted via ISO, cleared the metadata on the second disk, and then it successfully completed through commissioning and a ready state.

Wherever cloud-init is run it looks for disk meta data and prioritizes it based on what I saw.

[Blake Rouse (blake-rouse)]

It was recommended by smoser to add this to MAAS: http://paste.ubuntu.com/16683033/

But then it was discovered that will not handle all cases. A kernel parameter would be needed to force only one datasource.

[Blake Rouse (blake-rouse)]

Something like:

cloud-init-datasources=MAAS

[Scott Moser (smoser)]

cloud-init allows kernel command line to have yaml inline that contributes to cloud-config.
 http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/kernel-cmdline.txt

That woudl require an SRU to get the change functional though. As currently the kernel command line config is consumed after /etc/cloud/cloud.cfg but before /etc/cloud/cloud.cfg.d/* and it is /etc/cloud/cloud.cfg.d/90_dpkg.cfg that defines the standard list of datasources.

So, with a small SRUable change like: http://paste.ubuntu.com/16684929/

we could then have maas pass on the kernel command line a token like:
  cc:{'datasource_list': ['MAAS']}end_cc

basically that is:
  "cc:" + json.dumps({'datasource_list': ['MAAS']}) + "end_cc"

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.7~bzr1227-0ubuntu1

---------------
cloud-init (0.7.7~bzr1227-0ubuntu1) yakkety; urgency=medium

  * New upstream snapshot.
    - fix one more unit test to run inside buildd.

 -- Scott Moser <email address hidden> Sat, 04 Jun 2016 20:55:07 -0400

[Scott Moser (smoser)]

Hello,
An SRU upload of cloud-init for 16.04 that contains a fix for this bug has been made under bug 1595302. Please track that bug if you are interested.

[Scott Moser (smoser)]

fix is now released to xenial under bug 1595302. daily cloud-images with this newer version of cloud-init should appear in the next few days. Any image with a serial number *newer* than 20160707 should have cloud-init at 0.7.7~bzr1246-0ubuntu1~16.04.1 .

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.7

[Scott Moser (smoser)]

re-opening this for cloud-init as the change in cloud-init actually regressed the behavior.

[Scott Moser (smoser)]

This is now re-fixed in trunk (0b0f254a6935a1b1fff128fa177152dd519e1a3d).

[Scott Moser (smoser)]

This is fixed in cloud-init 0.7.9

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-0ubuntu1

---------------
cloud-init (0.7.9-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - release 0.7.9
    - doc: adjust headers in tests documentation for consistency.
    - integration test: initial commit of integration test framework
      [Wesley Wiedenmeier]
    - LICENSE: Allow dual licensing GPL-3 or Apache 2.0 [Jon Grimm]
    - Fix config order of precedence, putting kernel command line over system.
      [Wesley Wiedenmeier] (LP: #1582323)
    - Update the list of valid ssh keys. [Michael Felt]
    - network: add ENI unit test for statically rendered routes.
    - set_hostname: avoid erroneously appending domain to fqdn
      [Lars Kellogg-Stedman] (LP: #1647910)
    - doc: change 'nobootwait' to 'nofail' in docs [Anhad Jai Singh]
    - Replace an expired bit.ly link in code comment.

 -- Scott Moser <email address hidden> Fri, 23 Dec 2016 12:54:50 -0500

[Andres Rodriguez (andreserl)]

@Scott,

Any ETA for SRU?

[Bob Wise (countspongebob)]

I opened a bug (marked as duplicate) here with my details.

https://bugs.launchpad.net/maas/+bug/1645872

I just upgraded to 2.1.3 and attempted to repro. Node commissioning still fails as before. Appears to be to be marked as fixed for 2.1.3. Not fixed.

[Blake Rouse (blake-rouse)]

Bob,

This is fixed in MAAS the issue is still in the Xenial images. That is why cloud-init is still marked Confirmed for Xenial as it has not been SRU'd.

[Jon Grimm (jgrimm)]

The fix has been uploaded to the Xenial queue, waiting on SRU team. Just FYI that this is on the way.

[Bob Wise (countspongebob)]

Thanks for the update... how would I check the Xenial queue status? I note that "dpkg -l
 grep cloud-init" shows no package on my system.

[Jon Grimm (jgrimm)]

Hi Bob, you'll see comments to this bug once it is available in
xenial-proposed, and then when it is consequently migrated into
xenial-updates.

On Tue, Jan 31, 2017 at 12:30 PM, Bob Wise <email address hidden> wrote:

> Thanks for the update... how would I check the Xenial queue status? I note
> that "dpkg -l
> grep cloud-init" shows no package on my system.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1582323
>
> Title:
> Commissioning fails when competing cloud metadata resides on disk
>
> Status in cloud-init:
> Fix Released
> Status in MAAS:
> Fix Committed
> Status in MAAS 2.1 series:
> Fix Released
> Status in MAAS trunk series:
> Fix Committed
> Status in cloud-init package in Ubuntu:
> Fix Released
> Status in cloud-init source package in Xenial:
> Confirmed
>
> Bug description:
> A customer reused hardware that had previously deployed a RHEL
> Overcloud-controller which places metadata on the disk as a legitimate
> source, that cloud-init looks at by default. When the newly enlisted
> node appeared it had the name of "overcloud-controller-0" vs. maas-
> enlist, pulled from the disk metadata which had overridden MAAS'
> metadata. Commissioning continually failed on all of the nodes until
> the disk metadata was manually removed (KVM boot Ubuntu ISO, rm -f
> data or dd zeros to disk).
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1582323/+subscriptions
>

--
Jon Grimm
Engineering Manager, Ubuntu Server
Canonical Ltd

[Brian Murray (brian-murray)]

Thanks for uploading the fix for this bug report to -proposed. However, when reviewing the package in -proposed and the details of this bug report I noticed that the bug description is missing information required for the SRU process. You can find full details at http://wiki.ubuntu.com/StableReleaseUpdates#Procedure but essentially this bug is missing some of the following: a statement of impact, a test case and details regarding the regression potential. Thanks in advance!

[Scott Moser (smoser)]

Brian,
SRU template added. Sorry.

[Brian Murray (brian-murray)]

Hello M.Morana, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello M.Morana, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello M.Morana, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

In order to verify this on xenial, we'll configure a container to
have cloud-config like MAAS uses on the init command's command line.
Basically, instead of invokin /sbin/init for pid 1, we will use:

/sbin/init cc:{'datasource_list':['MAAS']}end_cc

This is equivalent to booting a full system with the above on the kernel
command line. When inside a container, cloud-init uses pid 1's cmdline
(/proc/1/cmdline) rather than the kernel's cmdline (/proc/cmdline).

Because of this bug, the system will find the NoCloud datasource that
lxc provides even though we told it on the command line to only look at
MAAS. Then, we will update the system with proposed, clean the system
to mimic a pristine "first boot" and reboot.

The second time, the system will fail to find the NoCloud datasource
becaus it correctly responded to the command line arguments telling it to
only look for MAAS datasource.

$ release=xenial
$ name="$release-test"
$ lxc init ubuntu-daily:$release $name

# change pid 1's command line, which is used by cloud-init
# instead of kernel command line inside a container.
$ cmdline="cc:{'datasource_list': ['MAAS']}end_cc"
$ lxc config set $name raw.lxc "lxc.init_cmd=/sbin/init $cmdline"

$ lxc start $name
# sleep
$ sleep 10

## show info about instance
$ lxc exec $name -- dpkg-query --show cloud-init
cloud-init 0.7.8-49-g9e904bb-0ubuntu1~16.04.4

# pid1 has the cmdline shown
$ lxc exec $name -- cat /proc/1/cmdline; echo
/sbin/initcc:{'datasource_list':['MAAS']}end_cc

$ cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": "DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]",
  "errors": []
 }
}

$ lxc file pull $name/etc/cloud/build.info -
build_name: server
serial: 20170209

## enable proposed, update
$ m=http://archive.ubuntu.com/ubuntu;
$ echo "deb $m $release-proposed main" | lxc file push - $name/etc/apt/sources.list.d/proposed.list
$ lxc exec $name -- sh -c 'apt-get update -q && apt-get install cloud-init' </dev/null
$ lxc exec $name -- dpkg-query --show cloud-init
cloud-init 0.7.9-0ubuntu1~16.04.2

# clean up to make it look like first boot.
$ lxc exec $name -- sh -c 'cd /var/lib/cloud; for d in *; do [ "$d" = "seed" ] || rm -vRf "$d"; done'
$ lxc exec $name -- sh -c 'rm -f /var/log/cloud*'
$ lxc restart $name
$ sleep 10

$ lxc exec $name -- cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": null,
  "errors": [
   "No instance datasource found.",
   "Can not apply stage config, no datasource found! Likely bad things to come!",
   "Can not apply stage final, no datasource found! Likely bad things to come!"
  ]
 }
}

$ lxc exec $name -- grep -i "Searching for" /var/log/cloud-init.log 2017-02-10 05:23:16,420 - __init__.py[DEBUG]: Searching for local data source in: []
2017-02-10 05:23:16,841 - handlers.py[DEBUG]: finish: init-local: SUCCESS: searching for local datasources
2017-02-10 05:23:19,358 - __init__.py[DEBUG]: Searching for network data source in: ['DataSourceMAAS']

[Scott Moser (smoser)]

$ name="$release-test"
$ lxc init ubuntu-daily:$release $name

# change pid 1's command line, which is used by cloud-init
# instead of kernel command line inside a container.
$ cmdline="cc:{'datasource_list': ['MAAS']}end_cc"
$ lxc config set $name raw.lxc "lxc.init_cmd=/sbin/init $cmdline"

$ lxc start $name
# sleep
$ sleep 10

## show info about instance
$ lxc exec $name -- dpkg-query --show cloud-init
cloud-init 0.7.8-68-gca3ae67-0ubuntu1~16.10.1

# pid1 has the cmdline shown
$ lxc exec $name -- cat /proc/1/cmdline; echo
/sbin/initcc:{'datasource_list':['MAAS']}end_cc

$ lxc exec $name -- cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": "DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]",
  "errors": []
 }
}

$ lxc file pull $name/etc/cloud/build.info -
build_name: server
serial: 20170207

## enable proposed, update
$ m=http://archive.ubuntu.com/ubuntu;
$ echo "deb $m $release-proposed main" | lxc file push - $name/etc/apt/sources.list.d/proposed.list
$ lxc exec $name -- sh -c 'apt-get update -q && apt-get install cloud-init' </dev/null
$ lxc exec $name -- dpkg-query --show cloud-init
cloud-init 0.7.9-0ubuntu1~16.10.1

# clean up to make it look like first boot.
$ lxc exec $name -- sh -c 'cd /var/lib/cloud; for d in *; do [ "$d" = "seed" ] || rm -vRf "$d"; done'
$ lxc exec $name -- sh -c 'rm -f /var/log/cloud*'
$ lxc restart $name
$ sleep 10

$ lxc exec $name -- cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": null,
  "errors": [
   "No instance datasource found.",
   "Can not apply stage config, no datasource found! Likely bad things to come!",
   "Can not apply stage final, no datasource found! Likely bad things to come!"
  ]
 }
}

$ lxc exec $name -- grep -i "Searching for" /var/log/cloud-init.log
2017-02-10 14:57:42,046 - __init__.py[DEBUG]: Searching for local data source in: []
2017-02-10 14:57:42,074 - handlers.py[DEBUG]: finish: init-local: SUCCESS: searching for local datasources
2017-02-10 14:57:43,805 - __init__.py[DEBUG]: Searching for network data source in: ['DataSourceMAAS']
2017-02-10 14:57:43,805 - handlers.py[DEBUG]: start: init-network/search-MAAS: searching for network data from DataSourceMAAS
2017-02-10 14:57:43,832 - handlers.py[DEBUG]: finish: init-network: SUCCESS: searching for network datasources

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/copyright: update License field to include Apache-2.0
  * debian/update-grub-legacy-ec2: fix to include kernels whose config
    has CONFIG_XEN=y (LP: #1379080).
  * debian/update-grub-legacy-ec2: detect kernels ending in -aws as
    ec2 bootable (LP: #1655934).
  * New upstream release.
    - doc: adjust headers in tests documentation for consistency.
    - pep8: fix issue found in zesty build with pycodestyle.
    - integration test: initial commit of integration test framework
      [Wesley Wiedenmeier]
    - LICENSE: Allow dual licensing GPL-3 or Apache 2.0 [Jon Grimm]
    - Fix config order of precedence, putting kernel command line over system.
      [Wesley Wiedenmeier] (LP: #1582323)
    - pep8: whitespace fix
    - Update the list of valid ssh keys. [Michael Felt]
    - network: add ENI unit test for statically rendered routes.
    - set_hostname: avoid erroneously appending domain to fqdn
      [Lars Kellogg-Stedman] (LP: #1647910)
    - doc: change 'nobootwait' to 'nofail' in docs [Anhad Jai Singh]
    - Replace an expired bit.ly link in code comment. [Joshua Harlow]

 -- Scott Moser <email address hidden> Tue, 31 Jan 2017 21:02:28 -0500

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-0ubuntu1~16.04.2

---------------
cloud-init (0.7.9-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * debian/update-grub-legacy-ec2: fix shell syntax error. (LP: #1662221)

cloud-init (0.7.9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/copyright: update License field to include Apache.
  * debian/update-grub-legacy-ec2: fix to include kernels whose config
    has CONFIG_XEN=y (LP: #1379080).
  * debian/patches/azure-use-walinux-agent.patch: continue relying on
    walinux agent in stable release.
  * New upstream release.
    - doc: adjust headers in tests documentation for consistency.
    - pep8: fix issue found in zesty build with pycodestyle.
    - integration test: initial commit of integration test framework
      [Wesley Wiedenmeier]
    - LICENSE: Allow dual licensing GPL-3 or Apache 2.0 [Jon Grimm]
    - Fix config order of precedence, putting kernel command line over system.
      [Wesley Wiedenmeier] (LP: #1582323)
    - pep8: whitespace fix [Scott Moser]
    - Update the list of valid ssh keys. [Michael Felt]
    - network: add ENI unit test for statically rendered routes.
    - set_hostname: avoid erroneously appending domain to fqdn
      [Lars Kellogg-Stedman] (LP: #1647910)
    - doc: change 'nobootwait' to 'nofail' in docs [Anhad Jai Singh]
    - Replace an expired bit.ly link in code comment. [Joshua Harlow]
    - user-groups: fix bug when groups was provided as string and had spaces
      [Scott Moser] (LP: #1354694)
    - when adding a user, strip whitespace from group list
      [Lars Kellogg-Stedman] (LP: #1354694)
    - fix decoding of utf-8 chars in yaml test
    - Replace usage of sys_netdev_info with read_sys_net
      [Joshua Harlow] (LP: #1625766)
    - fix problems found in python2.6 test. [Joshua Harlow]
    - Just use file logging by default [Joshua Harlow] (LP: #1643990)
    - Improve formatting for ProcessExecutionError [Wesley Wiedenmeier]
    - flake8: fix trailing white space
    - Doc: various documentation fixes [Sean Bright]
    - cloudinit/config/cc_rh_subscription.py: Remove repos before adding
      [Brent Baude]
    - packages/redhat: fix rpm spec file.
    - main: set TZ in environment if not already set. [Ryan Harper]

 -- Scott Moser <email address hidden> Mon, 06 Feb 2017 16:18:28 -0500

[DesktopMan (christian-auby)]

Is this fix available in the 16.04 LTS image used by MaaS? Commissioning fails for me and I'm wondering if this is the reason.

"Can not apply stage config, no datasource found! Likely bad things to come!"

"Failed to start Apply the settings specified in cloud-config"

[Bob Wise (countspongebob)]

I performed the latest updates and much to my dismay it did not resolve the issue.

On a whim, putting the node that failed to commission into recovery mode and then taking back out again resolved it.

So, there is at least a workaround now from the UI that will get a node into the system, but I would not consider the problem fully resolved.

[Felipe Alfaro Solana (felipe-alfaro-gmail)]

I'm experiencing exactly the same problem. Seems to be something weird with cloud-init that hasn't been fixed in 16.04.3 LTS.

[Deepa (dpaclt)]

I am facing same issue with very latest verion of MaaS MAAS version: 2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)

[Scott Moser (smoser)]

@Deepa,
Please file a new bug.
If possible, get
a.) a serial console log of the boot of the system
b.) ssh into the commissioning mode and run 'cloud-init collect-logs'.

thanks.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2668