Cloudinit modules should provide schema validation to better alert consumers to unsupported config

Bug #1692916 reported by Chad Smith on 2017-05-23
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Chad Smith
cloud-init (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Zesty
Medium
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
New feature to validate and log invalid schema warnings from cc_ntp cloud-config module.

[Test Case]
if [ ! -f lxc-proposed-snapshot ]; then
  wget https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/plain/bin/lxc-proposed-snapshot
  chmod 755 lxc-proposed-snapshot
fi

cat <<EOF > valid.conf
#cloud-config
ntp:
EOF
cat <<EOF > invalid.conf
#cloud-config
ntp: -1
EOF
for release in xenial zesty; do
        ref=$release-proposed;
        echo "$release START --------------";
        ./lxc-proposed-snapshot --proposed --publish $release $ref;
        lxc init $ref test-$release;
        lxc start test-$release;
        sleep 10;
        lxc exec test-$release -- sudo apt update;
        lxc file push valid.conf test-$release/valid.conf;
        lxc file push invalid.conf test-$release/invalid.conf;
        lxc exec test-$release -- python3 /usr/lib/python3/dist-packages/cloudinit/config/schema.py -c /valid.conf | grep Valid;
        lxc exec test-$release -- apt-cache depends cloud-init | grep jsonschema # should be empty;
        # invalid.conf will not generate errors when jsonschema is not present
        lxc exec test-$release -- python3 /usr/lib/python3/dist-packages/cloudinit/config/schema.py -c /invalid.conf | grep invalid;
        lxc exec test-$release -- sudo apt install python3-jsonschema
        # invalid.conf *will* generate errors when jsonschema is not present
         lxc exec test-$release -- python3 /usr/lib/python3/dist-packages/cloudinit/config/schema.py -c /invalid.conf | grep invalid;
done;

[Regression Potential]
We don't want to introduce a mandatory jsonschema dependency in older series.
Validate that older releases can run without errors when jsonschema is *not* installed.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=0a448dd034

=== End SRU Template ===

cloudinit needs a mechanism to parse and validate a strict schema definition for modules that parse user created #cloud-config yaml files.

Related branches

Scott Moser (smoser) on 2017-05-25
Changed in cloud-init:
importance: Undecided → Medium
Chad Smith (chad.smith) wrote :

potential valid and invalid ntp cloud-config files

Scott Moser (smoser) on 2017-07-31
Changed in cloud-init (Ubuntu):
status: New → Fix Released
Changed in cloud-init:
status: In Progress → Fix Committed
Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
Chad Smith (chad.smith) on 2017-08-01
description: updated
Chad Smith (chad.smith) on 2017-08-01
description: updated
Chad Smith (chad.smith) on 2017-08-01
description: updated
Changed in cloud-init:
status: Fix Committed → Fix Released
Scott Moser (smoser) on 2017-08-04
Changed in cloud-init:
status: Fix Released → Fix Committed
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu Zesty):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Zesty):
importance: Undecided → Medium

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in cloud-init (Ubuntu Zesty):
status: Confirmed → Fix Committed
tags: added: verification-needed-zesty
Chris J Arges (arges) wrote :

Hello Chad, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-233-ge586fe35-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Chad Smith (chad.smith) on 2017-08-28
description: updated
Chad Smith (chad.smith) wrote :

Validating xenial and zesty with the above script:
xenial START --------------
Creating xenial-proposed-123719471
...
The following packages will be upgraded:
  cloud-init
1 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 324 kB of archives.
After this operation, 44.0 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 cloud-init all 0.7.9-233-ge586fe35-0ubuntu1~16.04.1 [324 kB]
...
Valid cloud-config file /valid.conf
Valid cloud-config file /invalid.conf
...
The following NEW packages will be installed:
  python3-jsonschema
# And we now can see schema errors when jsonschema is installed
Cloud config schema errors: ntp.pools: -1 is not of type 'array'

zesty START --------------
Creating zesty-proposed-34023523
...
The following packages will be upgraded:
  cloud-init
...
Get:1 http://archive.ubuntu.com/ubuntu zesty-proposed/main amd64 cloud-init all 0.7.9-233-ge586fe35-0ubuntu1~17.04.1 [321 kB]
...
Creating test-zesty
Valid cloud-config file /valid.conf
Valid cloud-config file /invalid.conf
...
The following NEW packages will be installed:
  python3-jsonschema
# And we now can see schema errors when jsonschema is installed
Cloud config schema errors: ntp.pools: -1 is not of type 'array'

tags: added: verification-done-xenial verification-done-zesty
removed: verification-needed verification-needed-xenial verification-needed-zesty
description: updated
Chad Smith (chad.smith) on 2017-08-28
description: updated
Launchpad Janitor (janitor) wrote :
Download full text (6.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
    + debian/patches/cpick-5fb49bac-azure-identify-platform...
    + debian/patches/cpick-003c6678-net-remove-systemd-link...
    + debian/patches/cpick-1cd4323b-azure-remove-accidental...
    + debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
    + debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
    add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
    - cloudinit.net: add initialize_network_device function and tests
      [Chad Smith]
    - makefile: fix ci-deps-ubuntu target [Chad Smith]
    - tests: adjust locale integration test to parse default locale.
    - tests: remove 'yakkety' from releases as it is EOL.
    - centos: do not package systemd-fsck drop-in.
    - systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
    - tests: Add initial tests for EC2 and improve a docstring.
    - locale: Do not re-run locale-gen if provided locale is system default.
    - archlinux: fix set hostname usage of write_file. [Joshua Powers]
    - sysconfig: support subnet type of 'manual'.
    - tools/run-centos: make running with no argument show help.
    - Drop rand_str() usage in DNS redirection detection
      [Bob Aman] (LP: #1088611)
    - sysconfig: use MACADDR on bonds/bridges to configure mac_address
      [Ryan Harper]
    - net: eni route rendering missed ipv6 default route config
      [Ryan Harper] (LP: #1701097)
    - sysconfig: enable mtu set per subnet, including ipv6 mtu
      [Ryan Harper]
    - sysconfig: handle manual type subnets [Ryan Harper]
    - sysconfig: fix ipv6 gateway routes [Ryan Harper]
    - sysconfig: fix rendering of bond, bridge and vlan types.
      [Ryan Harper]
    - Templatize systemd unit files for cross distro deltas. [Ryan Harper]
    - sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
    - net: fix renaming of nics to support mac addresses written in upper
      case. (LP: #1705147)
    - tests: fixes for issues uncovered when moving to python 3.6.
    - sysconfig: include GATEWAY value if set in subnet
      [Ryan Harper]
    - Scaleway: add datasource with user and vendor data for Scaleway.
      [Julien Castets]
    - Support comments in content read by load_shell_content.
    - cloudinitlocal fail to run during boot [Hongjiang Zhang]
    - doc: fix disk setup example table_type options [Sandor Zeestraten]
    - tools: Fix exception handling. [Joonas Kylmälä]
    - tests: fix usage of mock in GCE test.
    - test_gce: Fix invalid mock of platform_reports_gce to return False
      [Chad Smith]
    - test: fix incorrect keyid for apt repository. [Joshua Powers]
    - tests: Update version of pylxd [Joshua Powers]
    - write_files: Remove log from helper function signatures.
      [Andrew Jorgensen]
    - doc: document...

Read more...

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :
Download full text (6.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-233-ge586fe35-0ubuntu1~17.04.1

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~17.04.1) zesty; urgency=medium

  * debian/cloud-init.templates: enable Scaleway cloud.
  * debian/cloud-init.templates: enable Aliyun cloud.
  * drop the following cherry picks, now incorporated in snapshot.
    + debian/patches/cpick-5fb49bac-azure-identify-platform...
    + debian/patches/cpick-003c6678-net-remove-systemd-link...
    + debian/patches/cpick-1cd4323b-azure-remove-accidental...
    + debian/patches/cpick-ebc9ecbc-Azure-Add-network-config...
    + debian/patches/cpick-11121fe4-systemd-make-cloud-final...
  * debian/patches/stable-release-no-jsonschema-dep.patch:
    add patch to remove optional dependency on jsonschema.
  * New upstream snapshot.
    - cloudinit.net: add initialize_network_device function and tests
      [Chad Smith]
    - makefile: fix ci-deps-ubuntu target [Chad Smith]
    - tests: adjust locale integration test to parse default locale.
    - tests: remove 'yakkety' from releases as it is EOL.
    - centos: do not package systemd-fsck drop-in.
    - systemd: make systemd-fsck run after cloud-init.service (LP: #1691489)
    - tests: Add initial tests for EC2 and improve a docstring.
    - locale: Do not re-run locale-gen if provided locale is system default.
    - archlinux: fix set hostname usage of write_file. [Joshua Powers]
    - sysconfig: support subnet type of 'manual'.
    - tools/run-centos: make running with no argument show help.
    - Drop rand_str() usage in DNS redirection detection
      [Bob Aman] (LP: #1088611)
    - sysconfig: use MACADDR on bonds/bridges to configure mac_address
      [Ryan Harper]
    - net: eni route rendering missed ipv6 default route config
      [Ryan Harper] (LP: #1701097)
    - sysconfig: enable mtu set per subnet, including ipv6 mtu
      [Ryan Harper]
    - sysconfig: handle manual type subnets [Ryan Harper]
    - sysconfig: fix ipv6 gateway routes [Ryan Harper]
    - sysconfig: fix rendering of bond, bridge and vlan types.
      [Ryan Harper]
    - Templatize systemd unit files for cross distro deltas. [Ryan Harper]
    - sysconfig: ipv6 and default gateway fixes. [Ryan Harper]
    - net: fix renaming of nics to support mac addresses written in upper
      case. (LP: #1705147)
    - tests: fixes for issues uncovered when moving to python 3.6.
    - sysconfig: include GATEWAY value if set in subnet
      [Ryan Harper]
    - Scaleway: add datasource with user and vendor data for Scaleway.
      [Julien Castets]
    - Support comments in content read by load_shell_content.
    - cloudinitlocal fail to run during boot [Hongjiang Zhang]
    - doc: fix disk setup example table_type options [Sandor Zeestraten]
    - tools: Fix exception handling. [Joonas Kylmälä]
    - tests: fix usage of mock in GCE test.
    - test_gce: Fix invalid mock of platform_reports_gce to return False
      [Chad Smith]
    - test: fix incorrect keyid for apt repository. [Joshua Powers]
    - tests: Update version of pylxd [Joshua Powers]
    - write_files: Remove log from helper function signatures.
      [Andrew Jorgensen]
    - doc: document the cmdli...

Read more...

Changed in cloud-init (Ubuntu Zesty):
status: Fix Committed → Fix Released

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments