address field not set when reading cmdline/initramfs configured network

Bug #1691135 reported by Scott Moser on 2017-05-16
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Unassigned
cloud-init (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Yakkety
Medium
Unassigned
Zesty
Medium
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
When cloud-init sees 'ip=' on the kernel command line, it will take
the networking configuration that is written by the ipconfig tool,
and convert it into the proper network config for the system
(/etc/network/interfaces).

This works well for the 'ip=dhcp' and 'ip=dhcp6' cases, but did not
work correctly for the "statick" path with a command line like:
   ip=<client-ip>::<gw-ip>:<netmask>:<hostname>:device:...

Cloud-init would stack trace when trying to bring up this networking
resulting in a system that did not boot properly.

[Test Case]
The basic idea below is:
 a.) launch an instance with proposed version of cloud-init.
 b.) inside instance, use cloud-init's net library to convert
     'net-eth1.cfg' into a different format, and the render that format
     using cloud-init's trunk tool 'net-convert.py'

## launch an instance.
$ release=xenial
$ ref=$release-proposed
$ lxc-proposed-snapshot --proposed --publish $release $ref
$ lxc launch $ref $name
$ lxc exec $ref $name /bin/bash

## get render tool
% wget https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1691135/+attachment/4888001/+files/bug-1691135.sh -O bug-1691135.sh
% sh ./bug-1691135.sh

## if that runs to completion, then it will show the /etc/network/interfaces
## file and the netplan etc/netplan/50-cloud-init.yaml file.
## if it fails it will stack trace.

As seen in the upstream commit, the change is in a very focused path,
only changing something when the protocol was 'static'.

[Regression Potential]
Potential for regression would be limited to the 'ip=' command line path.

[Other Info]
Upstream commit at
  https://git.launchpad.net/cloud-init/commit/?id=9d437489b

=== End SRU Template ===

As reported by user niluje, if the initramfs tools write a config that has static networking, then cloud-init would not populate the address field in its rendered network config (_klibc_to_config_entry).

The result would be incomplete or invalid configuration, and attempt to render that configuration would result in stacktrace due to KeyError.

This can be seen in trunk with the net-convert tool as below. Editing the yaml and uncommenting the 'address' line will work.

$ cat /tmp/foo.yaml
network:
  version: 1
  config:
   - name: eth1
     type: physical
     subnets:
      - 'type': 'static'
        'broadcast': '10.0.0.255'
        'control': 'manual'
        'gateway': '10.0.0.1'
        'dns_search': ['foo.com']
        'netmask': '255.255.255.0'
        'dns_nameservers': ['10.0.1.1']
# 'address': '10.0.0.2'
EOF

$ PYTHONPATH=$PWD ./tools/net-convert.py --network-data=/tmp/foo.yaml --kind=yaml --output-kind=eni -d /tmp/out.d
Input YAML
config:
- name: eth1
    subnets:
    - broadcast: 10.0.0.255
        control: manual
        dns_nameservers:
        - 10.0.1.1
        dns_search:
        - foo.com
        gateway: 10.0.0.1
        netmask: 255.255.255.0
        type: static
    type: physical
version: 1

Traceback (most recent call last):
  File "./tools/net-convert.py", line 82, in <module>
    main()
  File "./tools/net-convert.py", line 58, in main
    ns = network_state.parse_net_config_data(pre_ns)
  File "/home/smoser/src/cloud-init/cloud-init/cloudinit/net/network_state.py", line 42, in parse_net_config_data
    nsi.parse_config(skip_broken=skip_broken)
  File "/home/smoser/src/cloud-init/cloud-init/cloudinit/net/network_state.py", line 225, in parse_config
    self.parse_config_v1(skip_broken=skip_broken)
  File "/home/smoser/src/cloud-init/cloud-init/cloudinit/net/network_state.py", line 240, in parse_config_v1
    handler(self, command)
  File "/home/smoser/src/cloud-init/cloud-init/cloudinit/net/network_state.py", line 89, in decorator
    return func(self, command, *args, **kwargs)
  File "/home/smoser/src/cloud-init/cloud-init/cloudinit/net/network_state.py", line 296, in handle_physical
    if ':' in subnet['address']:
KeyError: 'address'

Related branches

Scott Moser (smoser) on 2017-05-16
Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser) on 2017-05-16
Changed in cloud-init:
status: Confirmed → Fix Committed
Scott Moser (smoser) wrote :

I'm attaching 'bug-show.sh' which actually shows this bug.
The net-eth1.cfg that is written is as 'ipconfig' from klibc would write for static config.
Then, we use the net-convert.py tool in cloud-init source to show the failure.

Scott Moser (smoser) on 2017-05-26
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → Fix Released
Changed in cloud-init (Ubuntu Xenial):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Yakkety):
status: New → Fix Committed
Changed in cloud-init (Ubuntu Zesty):
status: New → Confirmed
Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Confirmed
Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Confirmed
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Zesty):
importance: Undecided → Medium
Scott Moser (smoser) on 2017-06-02
description: updated
Scott Moser (smoser) wrote :
Scott Moser (smoser) on 2017-06-02
description: updated

Hello Scott, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Zesty):
status: Confirmed → Fix Committed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Yakkety):
status: Confirmed → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
Chad Smith (chad.smith) wrote :
Download full text (10.2 KiB)

Validated not stack traces on xenial yakkety and zesty:

Xenial --------------------------
--2017-06-16 17:13:03-- https://git.launchpad.net/~cloud-init-dev/cloud-init/plain/tools/net-convert.py
Resolving git.launchpad.net (git.launchpad.net)... 162.213.33.96, 162.213.33.95
Connecting to git.launchpad.net (git.launchpad.net)|162.213.33.96|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2683 (2.6K) [text/plain]
Saving to: ‘net-convert.py’

net-convert.py 100%[===================>] 2.62K --.-KB/s in 0s

2017-06-16 17:13:04 (254 MB/s) - ‘net-convert.py’ saved [2683/2683]

---- eni -> ./out-eni.d/ ----
Input YAML
config:
- name: eth1
    subnets:
    - address: 10.0.0.2
        broadcast: 10.0.0.255
        control: manual
        dns_nameservers:
        - 10.0.1.1
        dns_search:
        - foo.com
        gateway: 10.0.0.1
        netmask: 255.255.255.0
        type: static
    type: physical
version: 1

Internal State
!!python/object:cloudinit.net.network_state.NetworkState
_network_state:
    dns:
        nameservers: []
        search: []
    interfaces:
        eth1:
            address: null
            gateway: null
            inet: inet
            mac_address: null
            mode: manual
            mtu: null
            name: eth1
            subnets:
            - address: 10.0.0.2
                broadcast: 10.0.0.255
                control: manual
                dns_nameservers:
                - 10.0.1.1
                dns_search:
                - foo.com
                gateway: 10.0.0.1
                netmask: 255.255.255.0
                type: static
            type: physical
    routes: []
    use_ipv6: false
_version: 1
use_ipv6: false

---- netplan -> ./out-netplan.d/ ----
Input YAML
config:
- name: eth1
    subnets:
    - address: 10.0.0.2
        broadcast: 10.0.0.255
        control: manual
        dns_nameservers:
        - 10.0.1.1
        dns_search:
        - foo.com
        gateway: 10.0.0.1
        netmask: 255.255.255.0
        type: static
    type: physical
version: 1

Internal State
!!python/object:cloudinit.net.network_state.NetworkState
_network_state:
    dns:
        nameservers: []
        search: []
    interfaces:
        eth1:
            address: null
            gateway: null
            inet: inet
            mac_address: null
            mode: manual
            mtu: null
            name: eth1
            subnets:
            - address: 10.0.0.2
                broadcast: 10.0.0.255
                control: manual
                dns_nameservers:
                - 10.0.1.1
                dns_search:
                - foo.com
                gateway: 10.0.0.1
                netmask: 255.255.255.0
                type: static
            type: physical
    routes: []
    use_ipv6: false
_version: 1
use_ipv6: false

=== eni: /etc/network/interfaces ===
auto lo
iface lo inet loopback

# control-manual eth1
iface eth1 inet static
    address 10.0.0.2
    broadcast 10.0.0.255
    dns-nameservers 10.0.1.1
    dns-search foo.com
    gateway 10.0.0.1
    netmask 255.255.255.0

=== netplan: /etc/netplan/50-cloud-init.yaml ===

network:
...

tags: added: verification-done-xenial verification-done-yakkety verification-done-zesty
removed: verification-needed
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~17.04.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network c...

Read more...

Changed in cloud-init (Ubuntu Zesty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network...

Read more...

Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (3.4 KiB)

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network ...

Read more...

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers