ds-identify for Bigstep datasource checks wrong file

[Alex Sirbu]

=== Begin SRU Template ===
[Impact]
ds-identify would not identify a BigStep system the same way that
cloud-init would. The result would be a warning that ds-identify
didn't find a datasource.

[Test Case]
Launching an instance on Bigstep cloud is non-trivial for 16.04 or 16.10
as support is not yet present in their infrastructure.

So to test we will mimic Bigstep datasource.

lxc-proposed-snapshot is
  https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bin/lxc-proposed-snapshot
It publishes an image to lxd with proposed enabled and cloud-init upgraded.

# grab the address of lxcbr0
$ hostaddr=$(ifconfig lxcbr0 | awk '$1 == "inet" { print $2 }')
$ echo $hostaddr
10.0.3.1

$ echo '{"userdata_raw": null, "vendordata_raw": null, "metadata": {"local-hostname": "foohost", "instance-id": "abcdefg"}}' > bigstep.json
$ python -m SimpleHTTPServer 9999 &

$ release=xenial
$ ref=$release-proposed
$ lxc-proposed-snapshot --proposed --publish $release $ref
$ lxc init $ref $name
$ echo -n http://$hostaddr:9999/bigstep.json |
   lxc file push -p - $name/var/lib/cloud/data/seed/bigstep/url
$ echo 'datasource_list: [Bigstep, OpenStack, None]' |
   lxc file push -p - $name/etc/cloud/cloud.cfg.d/99-bigstep.cfg
$ lxc start $name
$ sleep 10
$ lxc file pull $name/run/cloud-init/result.json -

[Regression Potential]
Low. simply fixing a string.

[Other Info]
Upstream commit:
 https://git.launchpad.net/cloud-init/commit/?id=20a628c122583e15

=== End SRU Template ===

Currently, it checks that /var/lib/cloud/seed/bigstep/url exists.

However, the URL file is located at /var/lib/cloud/data/seed/bigstep/url.

Thus, the check should check the proper location.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-77-g4a2b2f87-0ubuntu1

---------------
cloud-init (0.7.9-77-g4a2b2f87-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - Fix filesystem creation when using "partition: auto" [Jonathan Ballet]
      (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive for
      nova-lxd. (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3.
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)

 -- Scott Moser <email address hidden> Fri, 24 Mar 2017 16:50:56 -0400

[Brian Murray (brian-murray)]

Hello Alex, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Alex, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

I would like to see some information about how the tests have been run and against which packages. As per the SRU procedures, each SRU verification is required to at least outline the steps that have been performed and what results have been recorded. Could you post those in the next comment?

[Chad Smith (chad.smith)]

Sorry forgot to past contents here on the testing. I had forgotten to click post comment when I finished testing per the described setup.

# Here's xenial which I just reran, yakkety to follow in about 10 mins BigStep was discovered in this case

csmith@fringe:~/src/sru$ echo $ref
xenial-proposed
csmith@fringe:~/src/sru$ lxc init $ref $name
Creating xp1
csmith@fringe:~/src/sru$ echo -n http://$hostaddr:9999/bigstep.json |
> lxc file push -p - $name/var/lib/cloud/data/seed/bigstep/url
csmith@fringe:~/src/sru$ echo 'datasource_list: [Bigstep, OpenStack, None]' |
> lxc file push -p - $name/etc/cloud/cloud.cfg.d/99-bigstep.cfg
csmith@fringe:~/src/sru$ lxc start $name
sleep 10csmith@fringe:~/src/sru$ sleep 10
10.249.77.167 - - [20/Apr/2017 10:10:51] "GET /bigstep.json HTTP/1.1" 200 -
csmith@fringe:~/src/sru$ lxc file pull $name/run/cloud-init/result.json -
{
 "v1": {
  "datasource": "DataSourceBigstep",
  "errors": []
 }

[Chad Smith (chad.smith)]

Same with yakety as above. replacing ref=yakkety-proposed and name=yp1

[Chad Smith (chad.smith)]

# yakkety done and Bigstep datasource detected

csmith@fringe:~/src/sru$ echo $ref
yakkety-proposed
csmith@fringe:~/src/sru$ echo $name
xp1
csmith@fringe:~/src/sru$ name=yp1
csmith@fringe:~/src/sru$ lxc init $ref $name
Creating yp1
csmith@fringe:~/src/sru$ echo -n http://$hostaddr:9999/bigstep.json |
> lxc file push -p - $name/var/lib/cloud/data/seed/bigstep/url
csmith@fringe:~/src/sru$ echo 'datasource_list: [Bigstep, OpenStack, None]' |
> lxc file push -p - $name/etc/cloud/cloud.cfg.d/99-bigstep.cfg
csmith@fringe:~/src/sru$ lxc start $name
slecsmith@fringe:~/src/sru$ sleep 10
10.249.77.36 - - [20/Apr/2017 10:25:37] "GET /bigstep.json HTTP/1.1" 200 -

csmith@fringe:~/src/sru$ lxc file pull $name/run/cloud-init/result.json -
{
 "v1": {
  "datasource": "DataSourceBigstep",
  "errors": []
 }
}

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773)
    - ...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-90-g61eb03fe-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: add Bigstep to list of sources. (LP: #1676460)
  * New upstream snapshot.
    - OpenStack: add 'dvs' to the list of physical link types. (LP: #1674946)
    - Fix bug that resulted in an attempt to rename bonds or vlans.
      (LP: #1669860)
    - tests: update OpenNebula and Digital Ocean to not rely on host
      interfaces.
    - net: in netplan renderer delete known image-builtin content.
      (LP: #1675576)
    - doc: correct grammar in capabilities.rst [David Tagatac]
    - ds-identify: fix detecting of maas datasource. (LP: #1677710)
    - netplan: remove debugging prints, add debug logging [Ryan Harper]
    - ds-identify: do not write None twice to datasource_list.
    - support resizing partition and rootfs on system booted without
      initramfs. [Steve Langasek] (LP: #1677376)
    - apt_configure: run only when needed. (LP: #1675185)
    - OpenStack: identify OpenStack by product 'OpenStack Compute'.
      (LP: #1675349)
    - GCE: Search GCE in ds-identify, consider serial number in check.
      (LP: #1674861)
    - Add support for setting hashed passwords [Tore S. Lonoy] (LP: #1570325)
    - Fix filesystem creation when using "partition: auto"
      [Jonathan Ballet] (LP: #1634678)
    - ConfigDrive: support reading config drive data from /config-drive.
      (LP: #1673411)
    - ds-identify: fix detection of Bigstep datasource. (LP: #1674766)
    - test: add running of pylint [Joshua Powers]
    - ds-identify: fix bug where filename expansion was left on.
    - advertise network config v2 support (NETWORK_CONFIG_V2) in features.
    - Bigstep: fix bug when executing in python3. [root]
    - Fix unit test when running in a system deployed with cloud-init.
    - Bounce network interface for Azure when using the built-in path.
      [Brent Baude] (LP: #1674685)
    - cloudinit.net: add network config v2 parsing and rendering [Ryan Harper]
    - net: Fix incorrect call to isfile [Joshua Powers] (LP: #1674317)
    - net: add renderers for automatically selecting the renderer.
    - doc: fix config drive doc with regard to unpartitioned disks.
      (LP: #1673818)
    - test: Adding integratiron test for password as list [Joshua Powers]
    - render_network_state: switch arguments around, do not require target
    - support 'loopback' as a device type.
    - Integration Testing: improve testcase subclassing [Wesley Wiedenmeier]
    - gitignore: adding doc/rtd_html [Joshua Powers]
    - doc: add instructions for running integration tests via tox.
      [Joshua Powers]
    - test: avoid differences in 'date' output due to daylight savings.
    - Fix chef config module in omnibus install. [Jeremy Melvin] (LP: #1583837)
    - Add feature flags to cloudinit.version. [Wesley Wiedenmeier]
    - tox: add a citest environment
    - Support chpasswd/list being a list in addition to a string.
      [Sergio Lystopad] (LP: #1665694)
    - doc: Fix configuration example for cc_set_passwords module.
      [Sergio Lystopad] (LP: #1665773...

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2834