Oracle: cloud-init openstack local detection too strict for oracle cloud

[Chad Smith]

=== Begin SRU Template ===
[Impact]
Cloud-init should detect OpenStack datasource on any OracleCloud instance.
Per the bug, it was falling through the DataSourceNone after upgrade to 18.3.9
and a system reboot.

[Test Case]
# Deploy an Oracle cloud bionic instance and validate upgrade/reboot path

cat > setup_proposed.sh <<EOF
#/bin/bash
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update -q;
apt-get install -qy cloud-init;
EOF

for vm in '129.146.86.46';
do
   echo '=== BEGIN ' $vm ' ==='
   ssh ubuntu@$vm grep CODENAME /etc/os-release;
   ssh ubuntu@$vm -- dpkg-query --show cloud-init;
   ssh ubuntu@$vm -- cloud-init status --long;
   ssh ubuntu@$vm -- cloud-init --version;
   ssh ubuntu@$vm -- cloud-init analyze show;
   scp setup_proposed.sh ubuntu@$vm:.;
   ssh ubuntu@$vm sudo bash ./setup_proposed.sh 2>&1 | egrep 'cloud-init';
   ssh ubuntu@$vm -- sudo cloud-init clean --logs --reboot;
   echo "After clean reboot, upgrade 18.3.9 should detect OpenStackLocal datasource"
   ssh-keygen -f ".ssh/known_hosts" -R $vm;
   ssh ubuntu@$vm -- cloud-init status --long;
   ssh ubuntu@$vm -- cloud-init --version;
done

[Regression Potential]
This fixes a critical issue in datasource detection on Oracle platforms only.
This should not regress any other platforms and is only a minor code path change
to include Oracle's DMI chassis asset tag as a valid OpenStack datasource type.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=0df2b42

=== End SRU Template ===

=== Original Description ===

cloud-init 18.3 cannot detect OpenStack datasource on Oracle cloud across reboots.

18.2 properly detects DataSourceOpenstack, but 18.3 added a detect_openstack helper function which fails to detect openstack datasource from the environment or DMI data matching a limited set of conditions:

  - DMI system-product_name in ['OpenStack Nova', 'OpenStack Compute']
      OracleCloud product-name is 'Standard PC (i440FX + PIIX, 1996)'
  - DMI chassis_asset_tag in ['OpenTelekomCloud']
         Oracle's chassis asset tag is 'OracleCloud.com'

  - proc/1/environ:product_name == 'OpenStack Nova'
       Oracle's /proc/1/environ has no product_name key:
$ sudo cat /proc/1/environ
HOME=/init=/sbin/initrecovery=TERM=linuxdrop_caps=BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-genericPATH=/sbin:/usr/sbin:/bin:/usr/binPWD=/rootmnt=/root

We need a more permissive detect_openstack function to detect Oracle during cloud-init's get_data() method.

[Server Team CI bot (server-team-bot)]

This bug is fixed with commit 3cee0bf8 to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=3cee0bf8

[Scott Moser (smoser)]

fixed in cosmic at 18.3-18-g3cee0bf8-0ubuntu1

[Chris Halse Rogers (raof)]

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.3-9-g2e62cb8a-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris Halse Rogers (raof)]

Hello Chad, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.3-9-g2e62cb8a-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chad Smith (chad.smith)]

SRU regresion validation logs for Oracle bionic and xenial

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 18.3-9-g2e62cb8a-0ubuntu1~18.04.2

---------------
cloud-init (18.3-9-g2e62cb8a-0ubuntu1~18.04.2) bionic-proposed; urgency=medium

  * cherry-pick 3cee0bf8: oracle: fix detect_openstack to report True on
    (LP: #1784685)

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~18.04.1) bionic-proposed; urgency=medium

  * New upstream snapshot. (LP: #1777912)
    - docs: note in rtd about avoiding /tmp when writing files
    - ubuntu,centos,debian: get_linux_distro to align with platform.dist
    - Fix boothook docs on environment variable name (INSTANCE_I ->
      INSTANCE_ID) [Marc Tamsky]
    - update_metadata: a datasource can support network re-config every boot
    - tests: drop salt-minion integration test
    - Retry on failed import of gpg receive keys.
    - tools: Fix run-container when neither source or binary package requested.
    - docs: Fix a small spelling error. [Oz N Tiram]
    - tox: use simplestreams from git repository rather than bzr.

cloud-init (18.3-0ubuntu1~18.04.1) bionic-proposed; urgency=medium

  * debian/rules: update version.version_string to contain packaged version.
    (LP: #1770712)
  * debian/patches/openstack-no-network-config.patch
    add patch to ignore Openstack network_config from network_data.json by
    default
  * New upstream release. (LP: #1777912)
    - release 18.3
    - docs: represent sudo:false in docs for user_groups config module
    - Explicitly prevent `sudo` access for user module [Jacob Bednarz]
    - lxd: Delete default network and detach device if lxd-init created them.
    - openstack: avoid unneeded metadata probe on non-openstack platforms
    - stages: fix tracebacks if a module stage is undefined or empty
      [Robert Schweikert]
    - Be more safe on string/bytes when writing multipart user-data to disk.
    - Fix get_proc_env for pids that have non-utf8 content in environment.
    - tests: fix salt_minion integration test on bionic and later
    - tests: provide human-readable integration test summary when --verbose
    - tests: skip chrony integration tests on lxd running artful or older
    - test: add optional --preserve-instance arg to integraiton tests
    - netplan: fix mtu if provided by network config for all rendered types
    - tests: remove pip install workarounds for pylxd, take upstream fix.
    - subp: support combine_capture argument.
    - tests: ordered tox dependencies for pylxd install
    - util: add get_linux_distro function to replace platform.dist
      [Robert Schweikert]
    - pyflakes: fix unused variable references identified by pyflakes 2.0.0.
    - - Do not use the systemd_prefix macro, not available in this environment
      [Robert Schweikert]
    - doc: Add config info to ec2, openstack and cloudstack datasource docs
    - Enable SmartOS network metadata to work with netplan via per-subnet
      routes [Dan McDonald]
    - openstack: Allow discovery in init-local using dhclient in a sandbox.
    - tests: Avoid using https in httpretty, improve HttPretty test case.
    - yaml_load/schema: Add invalid line and column nums to error message
    - Azure: Ignore NTFS mount errors when checking ephemeral drive
      [P...

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 18.3-9-g2e62cb8a-0ubuntu1~16.04.2

---------------
cloud-init (18.3-9-g2e62cb8a-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * cherry-pick 3cee0bf8: oracle: fix detect_openstack to report True on
    (LP: #1784685)

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot. (LP: #1777912)
    - docs: note in rtd about avoiding /tmp when writing files
    - ubuntu,centos,debian: get_linux_distro to align with platform.dist
    - Fix boothook docs on environment variable name (INSTANCE_I ->
      INSTANCE_ID) [Marc Tamsky]
    - update_metadata: a datasource can support network re-config every boot
    - tests: drop salt-minion integration test
    - Retry on failed import of gpg receive keys.
    - tools: Fix run-container when neither source or binary package requested.
    - docs: Fix a small spelling error. [Oz N Tiram]
    - tox: use simplestreams from git repository rather than bzr.

cloud-init (18.3-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/rules: update version.version_string to contain packaged version.
    (LP: #1770712)
  * debian/patches/openstack-no-network-config.patch
    add patch to ignore Openstack network_config from network_data.json by
    default
  * Refresh patches against upstream:
    + azure-use-walinux-agent.patch
    + ds-identify-behavior-xenial.patch
  * New upstream release. (LP: #1777912)
    - release 18.3
    - docs: represent sudo:false in docs for user_groups config module
    - Explicitly prevent `sudo` access for user module [Jacob Bednarz]
    - lxd: Delete default network and detach device if lxd-init created them.
    - openstack: avoid unneeded metadata probe on non-openstack platforms
    - stages: fix tracebacks if a module stage is undefined or empty
      [Robert Schweikert]
    - Be more safe on string/bytes when writing multipart user-data to disk.
    - Fix get_proc_env for pids that have non-utf8 content in environment.
    - tests: fix salt_minion integration test on bionic and later
    - tests: provide human-readable integration test summary when --verbose
    - tests: skip chrony integration tests on lxd running artful or older
    - test: add optional --preserve-instance arg to integraiton tests
    - netplan: fix mtu if provided by network config for all rendered types
    - tests: remove pip install workarounds for pylxd, take upstream fix.
    - subp: support combine_capture argument.
    - tests: ordered tox dependencies for pylxd install
    - util: add get_linux_distro function to replace platform.dist
      [Robert Schweikert]
    - pyflakes: fix unused variable references identified by pyflakes 2.0.0.
    - - Do not use the systemd_prefix macro, not available in this environment
      [Robert Schweikert]
    - doc: Add config info to ec2, openstack and cloudstack datasource docs
    - Enable SmartOS network metadata to work with netplan via per-subnet
      routes [Dan McDonald]
    - openstack: Allow discovery in init-local using dhclient in a sandbox.
    - tests: Avoid using https in httpretty, improve HttPretty test case.
    - yaml_load/schema: Add invalid ...

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in version 18.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3215