ds-identify does not find maas datasource

[Scott Moser]

=== Begin SRU Template ===
[Impact]
On Ubuntu core systems deployed with MAAS xenial and yakkety systems
would put a warning on the login screen stating that the datasource was
not found.

The issue only occurred on Ubuntu core. Ubuntu systems were not affected
as recent maas versions preseed cloud-init with 'datasource_list: [MAAS]'
which results in ds-identify just accepting the single entry as defined.

[Test Case]
The full test case involves
 * deploying through MAAS
 * enabling -proposed (without -proposed should show failure)
 * setting curtin config to show:
   system_upgrade: {enabled: True}}

[Regression Potential]
The changes that were done
 a.) renamed some variables to make code more readable
 b.) make searching for config less restrictive

due to 'a', there could be unintended bugs, but testing for
other datasources would likely have turned that up.

[Other Info]

=== End SRU Template ===

In ds-identify, the dscheck_MAAS calls check_config incorrectly, and as a result does not enable the MAAS datasource.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-87-gd23543eb-0ubuntu1

---------------
cloud-init (0.7.9-87-gd23543eb-0ubuntu1) zesty; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without initramfs.
      (LP: #1677376) [Steve Langasek]
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)

 -- Scott Moser <email address hidden> Thu, 30 Mar 2017 16:46:43 -0400

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andreas Hasenack (ahasenack)]

I'm having difficulties in reproducing the problematic case. I added this to /etc/maas/preseeds/curtin_userdata:

(...)
system_upgrade:
  enabled: True
late_commands:
(...)

Deploying yakkety without -proposed, I don't see any error regarding not finding a datasource, either in the logs, or in the console. Much less at the login screen, which just displays a prompt as usual.

Thinking it could have scrolled by too fast, I also deployed elsewhere where I had access to a serial console and could save all the output to a file, but also didn't see such an error there.

The console always shows this at the end:
(...)
[ 87.015604] cloud-init[2751]: Cloud-init v. 0.7.9 running 'modules:final' at Mon, 17 Apr 2017 15:14:24 +0000. Up 74.50 seconds.
[ 87.016119] cloud-init[2751]: Cloud-init v. 0.7.9 finished at Mon, 17 Apr 2017 15:14:36 +0000. Datasource DataSourceMAAS [http://10.96.0.10/MAAS/metadata/]. Up 86.09 seconds

Ubuntu 16.10 albany ttyS1

albany login:

Am I missing some condition to trigger the error? Could it be related to the MAAS version somehow? I tried with 2.1.5

[Scott Moser (smoser)]

I have verified deployment of nodes with maas 2.2 and upgrading the installed system to -proposed during the install.

that can be accomplished by adding the following to
 /etc/maas/preseeds/curtin_userdata_ubuntu

system_upgrade: {enabled: True}
apt:
  sources:
    proposed.list:
      source: deb $MIRROR $RELEASE-proposed main universe

[Andreas Hasenack (ahasenack)]

It's hard to reproduce the same error condition, as this happened during the development of a new feature (deploying ubuntu-core via MAAS). The closest I could get to it after some help from Ryan Harper was to run ds-identify on a normally deployed MAAS node with the current cloud-init and the proposed one, and check the results in /run/cloud-init/cloud.cfg.

Besides that, I also configured MAAS to enable the <release>-proposed pocket and the nodes deployed just fine, and with the new cloud-init from proposed installed.

For Yakkety:

With the current cloud-init, where the problem appears:
 *** 0.7.9-48-g1c795b9-0ubuntu1~16.10.1 500
        500 http://br.archive.ubuntu.com/ubuntu yakkety-updates/main amd64 Packages

# remove generated files
root@15-89:/run/cloud-init# rm cloud.cfg ds-identify.log

# cloud-init configuration, set via dpkg-reconfigure cloud-init and unchecking all items:
root@15-89:/run/cloud-init# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ ]

# maas datasource config file, written to by MAAS:
root@15-89:/run/cloud-init# cat /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg
# written by cloud-init debian package per preseed entries
# cloud-init/{maas-metadata-url,/maas-metadata-credentials}
datasource:
  MAAS: {consumer_key: xCkt8HsCeFKXBgm5SD, metadata_url: 'http://10.0.5.5:5240/MAAS/metadata/',
    token_key: 9pjmU6kjNAfdhe3xsJ, token_secret: kARCFjDaswVYDRLTSCTg9rrvXMjB7cGb}

# let's call ds-identify:
root@15-89:/run/cloud-init# unset DS_MAIN
root@15-89:/run/cloud-init# /usr/lib/cloud-init/ds-identify

# and we have no MAAS in cloud.cfg (somehow it thinks ec2 could be a candidate):
root@15-89:/run/cloud-init# cat cloud.cfg
datasource_list: [ Ec2, None ]
datasource: {Ec2: {strict_id: "warn"}}

# the ds-identify log file has
root@15-89:/run/cloud-init# cat ds-identify.log
(...)
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Ec2 OpenNebula OpenStack OVF SmartOS
(...)
is_container=false
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
1 datasources returned maybe: Ec2
[up 10525.51s] returning 0

Now I install the package from proposed:
 *** 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 500
        500 http://br.archive.ubuntu.com/ubuntu yakkety-proposed/main amd64 Packages

And repeat the steps. This time I get:
root@15-89:/run/cloud-init# cat cloud.cfg
datasource_list: [ MAAS, None ]

And the ds-identify log has:
root@15-89:/run/cloud-init# cat ds-identify.log
(...)
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Ec2 GCE OpenNebula OpenStack OVF SmartOS
(...)
is_container=false
check for 'MAAS' returned found
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
Found single datasource: MAAS
[up 10721.60s] returning 0

[Andreas Hasenack (ahasenack)]

For xenial:

With the current cloud-init, where the problem appears:
 *** 0.7.9-48-g1c795b9-0ubuntu1~16.04.1 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages

# remove generated files
root@15-89:/run/cloud-init# rm cloud.cfg ds-identify.log

# cloud-init configuration, set via dpkg-reconfigure cloud-init and unchecking all items:
root@15-89:/run/cloud-init# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ ]

# maas datasource config file, written to by MAAS:
root@15-89:/run/cloud-init# cat /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg
# written by cloud-init debian package per preseed entries
# cloud-init/{maas-metadata-url,/maas-metadata-credentials}
datasource:
  MAAS: {consumer_key: L8SxaFb29L3rzc3Vw8, metadata_url: 'http://10.0.5.5:5240/MAAS/metadata/',
    token_key: 92LGSD8NBTHbD7n7T8, token_secret: FAbznaWcx72ryaK6SrErnqeK2z9LH2Dj}

# let's call ds-identify:
root@15-89:/run/cloud-init# unset DS_MAIN
root@15-89:/run/cloud-init# /usr/lib/cloud-init/ds-identify

# and we have no MAAS in cloud.cfg (somehow it thinks ec2 could be a candidate):
root@15-89:/run/cloud-init# cat cloud.cfg
di_report:
  datasource_list: [ Ec2, None ]
  datasource: {Ec2: {strict_id: "warn"}}

# the ds-identify log file has
root@15-89:/run/cloud-init# cat ds-identify.log
(...)
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Ec2 OpenNebula OpenStack OVF SmartOS
(...)
is_container=false
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
1 datasources returned maybe: Ec2
[up 273.31s] returning 0

Now I install the package from proposed:
 *** 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages

90_dpkg.cfg still has an empty list:
root@15-89:/run/cloud-init# cat /etc/cloud/cloud.cfg.d/90_dpkg.cfg
# to update this file, run dpkg-reconfigure cloud-init
datasource_list: [ ]

And repeat the steps. This time I get:
root@15-89:/run/cloud-init# cat cloud.cfg
di_report:
  datasource_list: [ MAAS, None ]

And the ds-identify log has:
(...)
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean Ec2 GCE OpenNebula OpenStack OVF SmartOS
(...)
is_container=false
check for 'MAAS' returned found
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
Found single datasource: MAAS
[up 401.41s] returning 0

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2848