support strict cloud platform identification and warnings [ds-identify]

[Scott Moser]

=== Begin SRU Template ===
[Impact]
Cloud-init searches through each of its datasources until it finds one.
This can be slow or error prone leading to timeouts or other problems
in boot. It also makes cloud-init not suitable to be installed in an
image as it is very much not 'inert' even when it is not necessary.

The changes covered in this bug add code that runs in a systemd generator
that quickly determines which datasource should be used. That code
(ds-identify) can be configured to disable cloud-init if no source is found.
By doing so, no cloud-init python code is even executed making for faster
boot.

The changes in behavior in this first phase are:
 * 16.10 : Attempt a search. If a positive result is found, enable only
   that datasource in cloud-init. If no result is found, keep cloud-init
   enabled. cloud-init will search its normal list and issue warnings.

 * 16.04 : report only mode.
   Attempt a search, but in report only mode. Warnings are issued
   by cloud-init if it uses a datasource not in the result list.

[Test Case]
a.) forced fail
    1.) launch instance on KVM OpenStack. enable proposed, upgrade.
    2.) rm -Rf /var/lib/cloud /var/log/cloud-init* ;
    3.) modify /usr/lib/cloud-init/ds-identify to
           return $DS_NOT_FOUND
        in the first line of 'dscheck_OpenStack'
    4.) reboot
    5.) ssh in. You should see a warning on stderr on ssh connection.

b.) forced fail with warnings disabled.
    follow 'a' above, but launch instance with user-data of
      #cloud-config
      warnings:
        dsid_missing_source: off
    Upon ssh in in step 5, no warnings should be seen.

c.) normal instance
    1.) launch instance, enable proposed, upgrade
    2.) rm -Rf /var/lib/cloud /var/log/cloud-init*
    3.) reboot
    4.) expect that same datasource is found.

[Other Information]
See also: https://lists.ubuntu.com/archives/ubuntu-devel/2017-February/039697.html
=== End SRU Template ===

Related bugs:
 * bug 1660385: Alert user of Ec2 Datasource on lookalike cloud

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: cloud-init 0.7.9-38-g83606ae-0ubuntu1
ProcVersionSignature: Ubuntu 4.9.0-15.16-generic 4.9.5
Uname: Linux 4.9.0-15-generic x86_64
ApportVersion: 2.20.4-0ubuntu2
Architecture: amd64
Date: Fri Mar 3 04:25:51 2017
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-47-gc81ea53-0ubuntu1

---------------
cloud-init (0.7.9-47-gc81ea53-0ubuntu1) zesty; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
      login.
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - tools/ds-identify: look at cloud.cfg when looking for ec2 strict_id.
    - tools/ds-identify: fix bug that would mistakingly enable vmware guest
      customization.

 -- Scott Moser <email address hidden> Fri, 03 Mar 2017 01:56:19 -0500

[Chris Halse Rogers (raof)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris Halse Rogers (raof)]

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I've walked through the test case description above for xenial and verified it functioned as expected using 3 different instances on openstack cloud.

[Scott Moser (smoser)]

I have walked through the 3 test cases on 3 different instances in
an OpenStack installation with yakkety images.

There was a difference in behavior from what was listed in the description
for item 'b'. That is by design, and I just didn't realize it when writing
the test case.

On xenial, where ds-identify is set to 'report' mode, you will see the
warning. But on yakkety, where ds-identify is enabled, the system does
not warn as it goes on and uses the Ec2 metadata service. So instead
you see a warning that this system is using the Ec2 metadata service
and it should not be.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-xenial.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:34:10 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-yakkety.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:37:28 -0500

[kay (kay-diam)]

I use cloudinit for VM provisioning which doesn't use any cloud providers. I just put the data into the /mnt/var/lib/cloud/seed/nocloud directory.

And here is the message I get:

**************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# ['ConfigDrive', 'None'] #
# However, the datasource used was: NoCloud #
# #
# In the future, cloud-init will only attempt to use datasources that #
# are identified or specifically configured. #
# For more information see #
# https://bugs.launchpad.net/bugs/1669675 #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# After you have filed a bug, you can disable this warning by launching #
# your instance with the cloud-config below, or putting that content #
# into /etc/cloud/cloud.cfg.d/99-warnings.cfg #
# #
# #cloud-config #
# warnings: #
# dsid_missing_source: off #
**************************************************************************

Is it a bug? How can I resolve it without ignoring these warnings?

[Kurt Garloff (kgarloff)]

The OpenStack detection fails on the 1st gen compute hosts (using Xen hypervisor) in the Open Telekom Cloud.

Handle 0x0100, DMI type 1, 27 bytes
System Information
 Manufacturer: Xen
 Product Name: HVM domU
 Version: 4.1.2_115-908.762.
 Serial Number: 3c569b61-463d-49e0-ad09-c5e2235ce658
 UUID: 3C569B61-463D-49E0-AD09-C5E2235CE658
 Wake-up Type: Power Switch
 SKU Number: Not Specified
 Family: Not Specified

Handle 0x0300, DMI type 3, 13 bytes
Chassis Information
 Manufacturer: Xen
 Type: Other
 Lock: Not Present
 Version: Not Specified
 Serial Number: Not Specified
 Asset Tag: OpenTelekomCloud
 Boot-up State: Safe
 Power Supply State: Safe
 Thermal State: Safe
 Security Status: Unknown

Detection in ds-identify could look for Xen (HVM domU) and the Asset Tag.

[Kurt Garloff (kgarloff)]

... Alternatively, the UUID looks also suspiciously like OpenStack ...

[Scott Moser (smoser)]

Kurt, I've moved your comments into a new bug 1756471 and have asked a question there.

[jpp (jeanpierre-pilot)]

+filebug?.field.tags=dsid

[Oproma Inc. (opradm)]

Good Morning,

We have just received this error while trying to upgrade from Ubuntu 16.04.6 LTS to 18.04.2 LTS
```**************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# ['Ec2', 'None'] #
# However, the datasource used was: CloudStack #
# #
# In the future, cloud-init will only attempt to use datasources that #
# are identified or specifically configured. #
# For more information see #
# https://bugs.launchpad.net/bugs/1669675 #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# After you have filed a bug, you can disable this warning by launching #
# your instance with the cloud-config below, or putting that content #
# into /etc/cloud/cloud.cfg.d/99-warnings.cfg #
# #
# #cloud-config #
# warnings: #
# dsid_missing_source: off #
**************************************************************************

Disable the warnings above by:
  touch /home/pcutrone/.cloud-warnings.skip
or
  touch /var/lib/cloud/instance/warnings/.skip
```