Dsid missing source on Finnish provider Nebula and RDO installs

[Johan Ehnberg]

=== Begin SRU Template ===
[Impact]
When running on a Redhat RDO install (such as Finnish provider Nebula),
instances will show a warning on ssh login or in the cloud-init.log file.

This was by design, as we wanted to find places where ds-identify was
not working correctly.

[Test Case]
1. Launch an instance on a RDO deployed cloud
2. enable proposed, upgrade
3. rm -Rf /var/lib/cloud /var/log/cloud*
4. reboot
5. ssh back in, expect to see no warning.

[Regression Potential]
regression potential is low. The code change made here
was just to consider either 'OpenStack Nova' or 'OpenStack Compute'
as identifying openstack where previously only the 'Nova' string
was considered. This does provide a larger potential for false-positive
in identifying a non-openstack cloud as openstack, but that should
be seemingly small.

[Other Info]

=== End SRU Template ===

**************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# ['Ec2', 'None'] #
# However, the datasource used was: OpenStack #
...

[Scott Moser (smoser)]

Hi,
Thanks for taking the time to file this bug.
Could you please follow the instructiosn at
 https://gist.github.com/smoser/dd45b6324be937b67b80bac7aa1a6c47
and attach to this bug the resulting 'debug.tar.gz' file?

That will help us to find a fix for other users.

OpenStack has many different backends that can be used. The libvirt+qemu/kvm backend has been providing information in the smbios since the 2013.1 (Grizzly) release, which went end of life in March of 2014.

Other backends and architecture combinations do not identify themselves in a detectable way. I already have bugs open for:
 * bug 1669875 identify openstack vmware platform
 * bug 1663315 identify openstack kvm platform on ppc64
 * bug 1663304 identify openstack kvm platform on arm64
 * bug 1661797 identify lxd-nova platform to enable Openstack datasource

Thanks again for taking the time to open this bug.
Please feel free to follow up with any questions you might have.

Scott

[Scott Moser (smoser)]

Once you've attached the requested information, please set the bug to 'Confirmed' state.

[Johan Ehnberg (johan-ehnberg)]

WARNING to anyone using the script mentioned:
The script collects a lot of sensitive data even without --all. Looking at the tar file, I can see email logs, auth logs, service logs and more. I am sure these are completely unrelated to the bug. Run this in a test instance!

[Scott Moser (smoser)]

Johan, you're right, and thanks for that comment. I'll try to make that more clear.

[Scott Moser (smoser)]

From the collected debug.tar.gz, I see in the
 sys-class-dmi-id: product_name:OpenStack Compute

I'll broaden the check to accept that in addition to 'OpenStack Nova'.

[Scott Moser (smoser)]

Some poking around revealed to me that RDO's default install will contain 'OpenStack Nova' as the product name, which is what nebula.fi is showing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-87-gd23543eb-0ubuntu1

---------------
cloud-init (0.7.9-87-gd23543eb-0ubuntu1) zesty; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without initramfs.
      (LP: #1677376) [Steve Langasek]
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)

 -- Scott Moser <email address hidden> Thu, 30 Mar 2017 16:46:43 -0400

[Brian Murray (brian-murray)]

Hello Johan, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Johan, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Johan Ehnberg (johan-ehnberg)]

I can verify the fix. However,
    rm -Rf /var/lib/cloud /var/log/cloud*
will cause the SSH key to be reset, which is easily a bigger issue than getting a cloud-init warning.

[Piotr (piotrmuchowski)]

the patch from xenial-proposed works for me

I did:
sudo vi /etc/apt/sources.list # adding xenial-proposed
sudo apt-get update
sudo apt-get install cloud-init
sudo rm -Rf /var/lib/cloud /var/log/cloud*
sudo reboot

When I logged in second time, the ssh server key was different as it was regenerated again, but the message about datasources no longer appears.

[Piotr (piotrmuchowski)]

the version from xenial-proposed is :

ii cloud-init 0.7.9-90-g61eb03

[Scott Moser (smoser)]

Hi,
If someone here (Piotr?) could verify yakkety, that would be wonderful, and just post the feedback here.

[Scott Moser (smoser)]

I'm going to mark this verification-done-yakkety, based only on test of xenial and zesty. The code path is the same for yakkety as it is for the other two. I do not have access to a cloud that showed this issue and users who do generally run 16.04.

That said, if anyone *can* boot a yakkety with the new cloud-init, please do and report here.

[Piotr (piotrmuchowski)]

Sorry for delay, I have just checked on Yakkety as well

ubuntu@yakkety-test:~$ tail /etc/apt/sources.list
...
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu yakkety partner
# deb-src http://archive.canonical.com/ubuntu yakkety partner
deb http://archive.ubuntu.com/ubuntu/ yakkety-proposed restricted main multiverse universe

ubuntu@yakkety-test:~$

-----------

ubuntu@yakkety-test:~$ sudo apt-get install cloud-init
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  cloud-init
1 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.
Need to get 307 kB of archives.
After this operation, 38.9 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu yakkety-proposed/main amd64 cloud-init all 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 [307 kB]
Fetched 307 kB in 0s (334 kB/s)
Preconfiguring packages ...
(Reading database ... 85299 files and directories currently installed.)
Preparing to unpack .../cloud-init_0.7.9-90-g61eb03fe-0ubuntu1~16.10.1_all.deb ...
Unpacking cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) over (0.7.9-48-g1c795b9-0ubuntu1~16.10.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) ...
Leaving 'diversion of /etc/init/ureadahead.conf to /etc/init/ureadahead.conf.disabled by cloud-init'
ubuntu@yakkety-test:~$ sudo rm -Rf /var/lib/cloud /var/log/cloud*
ubuntu@yakkety-test:~$ dpkg-query -l cloud-init
...
ii cloud-init 0.7.9-90-g61eb03 all Init scripts for cloud instances
ubuntu@yakkety-test:~$

after reboot the "DSID" warning message is gone

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

[Piotr (piotrmuchowski)]

Thanks,

Just verified with Xenial and Yakkety cloud images.
(md5 aae5d19b4e9744e3d4d633ddb5ae6aae and cc117ce672a0d0ba1ad342c55f8c7010 respectively)

After applying full upgrade ( as a part of cloud-init post-config script ) warning is gone.

[Piotr (piotrmuchowski)]

proposed was not enabled in last test

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2840