cloud-init and nplan do not parse and use OpenStack networking correctly with netmask

[Dimitri John Ledkov]

=== Begin SRU Template ===
[Impact]
On Openstack instances, cloud-init incorrectly rendered netplan
configuration files. The result is that networking does not work
as expected.

Note that this is not a default configuration on any Ubuntu provided images.
Default images use ifupdown (eni) rendering which did not have this issue.

[Test Case]
The basic idea below is:
 a.) launch an instance with proposed version of cloud-init.
 b.) inside instance, get cloud-init's network rendering tool from trunk
 c.) run the rendering tool against a config that failed before.
 d.) check rendered netplan config to verify it has the correct format.
     The failed output would have 'addresses' with a format like:
     172.19.1.34/255.255.255.0
     The expected output would be 'cidr' format:
     172.19.1.34/24

## launch an instance.
$ release=xenial
$ ref=$release-proposed
$ lxc-proposed-snapshot --proposed --publish $release $ref
$ lxc init $ref $name

## get render tool
$ wget https://git.launchpad.net/~cloud-init-dev/cloud-init/plain/tools/net-convert.py -O net-convert.py

## write the network_data.json
$ cat >network_data.json <<EOF
{
 "links": [
  {"ethernet_mac_address": "aa:ab:ac:ad:ae:00",
   "id": "tap1a", "type": "phy", "vif_id": "1a81968a"}
 ],
 "networks": [
   {"id": "network0", "ip_address": "172.19.1.34", "link": "tap1a",
    "netmask": "255.255.255.0", "network_id": "dacd568d", "type": "ipv4",
    "routes": [
     {"gateway": "172.19.3.254", "netmask": "0.0.0.0",
      "network": "0.0.0.0"}]}
 ],
 "services": [{"address": "172.19.0.12", "type": "dns"}]
}
EOF

## run the converter
$ ./net-convert.py --network-data=network_data.json \
     --kind=network_data.json --output-kind=netplan \
     -m eth1,aa:ab:ac:ad:ae:00 --directory=./out.d

## check the output
$ cat out.d/etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth1:
            addresses:
            - 172.19.1.34/24
            match:
                macaddress: aa:ab:ac:ad:ae:00
            nameservers:
                addresses:
                - 172.19.0.12
            routes:
            - to: 0.0.0.0/0
                via: 172.19.3.254
            set-name: eth1

## show the cloud-init versions
$ dpkg-query --show cloud-init
...

[Regression Potential]
The change is fairly safe in that it basically renders:
  172.19.1.34/24
instead of
  172.19.1.34/255.255.255.0

The previous rendering just plain did not work as it is not valid
input for netplan. So the regression path there should be low.

The common code changes could shake out other failures in the networking
path.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=16a7302f6a

lxc-proposed-snapshot is
  https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bin/lxc-proposed-snapshot
It publishes an image to lxd with proposed enabled and cloud-init upgraded.
=== End SRU Template ===

networking data josn has:

"ip_address" : "104.130.20.155",
"netmask" : "255.255.255.0"

"ip_address" : "10.184.3.234",
"netmask" : "255.255.240.0",

that got rendered into nplan as:
 - 104.130.20.155/255.255.255.0
 - 10.184.3.234/255.255.240.0

which it failed to parse

Stderr: Error in network definition //etc/netplan/50-cloud-init.yaml line 32 column 12: invalid prefix length in address '104.130.20.155/255.255.255.0'

I believe nplan is expecing CIDR notation of /24 or some such. I belive the current plan is to fix cloud-init to generate /24 cidr notation in the nplan renderer.

This needs SRU into xenial.

[Steve Langasek (vorlon)]

We've discussed on IRC that netplan shouldn't accept /x.x.x.x syntax for netmasks and should require CIDR for simplicity/consistency. So this is on netplan to output the correct v2 yaml.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      [Scott Moser] (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - python2.6: fix unit tests usage of assertNone and format.
    - test: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - cc_ntp: Restructure cc_ntp unit tests. [Chad Smith] (LP: #1692794)

 -- Scott Moser <email address hidden> Fri, 26 May 2017 15:11:48 -0400

[Dimitri John Ledkov (xnox)]

Is SRU with this change in progress for xenial?

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chad Smith (chad.smith)]

Validated xenial yakkety and zesty render /# instead of /255.255.255.X
xenial START --------------

Internal State
!!python/object:cloudinit.net.network_state.NetworkState
_network_state:
    dns:
        nameservers:
        - 172.19.0.12
        search: []
    interfaces:
        eth1:
            address: null
            gateway: null
            inet: inet
            mac_address: aa:ab:ac:ad:ae:00
            mode: manual
            mtu: null
            name: eth1
            subnets:
            - address: 172.19.1.34
                ipv4: true
                netmask: 255.255.255.0
                routes:
                - gateway: 172.19.3.254
                    netmask: 0.0.0.0
                    network: 0.0.0.0
                type: static
            type: physical
    routes: []
    use_ipv6: false
_version: 1
use_ipv6: false

Checking out.d/etc/netplan/50-cloud-init.yaml ----

network:
    version: 2
    ethernets:
        eth1:
            addresses:
            - 172.19.1.34/24
            match:
                macaddress: aa:ab:ac:ad:ae:00
            nameservers:
                addresses:
                - 172.19.0.12
            routes:
            - to: 0.0.0.0/0
                via: 172.19.3.254
            set-name: eth1
cloud-init 0.7.9-153-g16a7302f-0ubuntu1~16.04.1
xenial DONE --------------
yakkety START --------------

Internal State
!!python/object:cloudinit.net.network_state.NetworkState
_network_state:
    dns:
        nameservers:
        - 172.19.0.12
        search: []
    interfaces:
        eth1:
            address: null
            gateway: null
            inet: inet
            mac_address: aa:ab:ac:ad:ae:00
            mode: manual
            mtu: null
            name: eth1
            subnets:
            - address: 172.19.1.34
                ipv4: true
                netmask: 255.255.255.0
                routes:
                - gateway: 172.19.3.254
                    netmask: 0.0.0.0
                    network: 0.0.0.0
                type: static
            type: physical
    routes: []
    use_ipv6: false
_version: 1
use_ipv6: false

Checking out.d/etc/netplan/50-cloud-init.yaml ----

network:
    version: 2
    ethernets:
        eth1:
            addresses:
            - 172.19.1.34/24
            match:
                macaddress: aa:ab:ac:ad:ae:00
            nameservers:
                addresses:
                - 172.19.0.12
            routes:
            - to: 0.0.0.0/0
                via: 172.19.3.254
            set-name: eth1
cloud-init 0.7.9-153-g16a7302f-0ubuntu1~16.10.1
yakkety DONE --------------
zesty START --------------

Internal State
!!python/object:cloudinit.net.network_state.NetworkState
_network_state:
    dns:
        nameservers:
        - 172.19.0.12
        search: []
    interfaces:
        eth1:
            address: null
            gateway: null
            inet: inet
            mac_address: aa:ab:ac:ad:ae:00
            mode: manual
            mtu: null
            name: eth1
            subnets:
            - address: 172.19.1.34
                ipv4: true
                netmask: 255.255.255.0
                routes:
              ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~17.04.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network c...

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network...

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network ...

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2882