netdev_pformat key error on FreeBSD with 18.3

[do3meli]

i am running cloud-init on commit id c42a926ae730994f66fe87c264b65f6e4dca69a1 against a FreeBSD 10.4 Host an getting the following stacktrace:

2018-07-02 11:40:18,158 - util.py[DEBUG]: Cloud-init v. 18.3 running 'init' at Mon, 02 Jul 2018 11:40:18 +0000. Up 20.11459589 seconds.
2018-07-02 11:40:18,159 - main.py[DEBUG]: No kernel command line url found.
2018-07-02 11:40:18,159 - main.py[DEBUG]: Closing stdin.
2018-07-02 11:40:18,172 - util.py[DEBUG]: Writing to /var/log/cloud-init.log - ab: [644] 0 bytes
2018-07-02 11:40:18,175 - util.py[DEBUG]: Changing the ownership of /var/log/cloud-init.log to 0:0
2018-07-02 11:40:18,175 - util.py[DEBUG]: Running command ['ifconfig', '-a'] with allowed return codes [0, 1] (shell=False, capture=True)
2018-07-02 11:40:18,195 - util.py[WARNING]: failed stage init
2018-07-02 11:40:18,196 - util.py[DEBUG]: failed stage init
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cloud_init-18.3-py2.7.egg/cloudinit/cmd/main.py", line 655, in status_wrapper
    ret = functor(name, args)
  File "/usr/local/lib/python2.7/site-packages/cloud_init-18.3-py2.7.egg/cloudinit/cmd/main.py", line 284, in main_init
    sys.stderr.write("%s\n" % (netinfo.debug_info()))
  File "/usr/local/lib/python2.7/site-packages/cloud_init-18.3-py2.7.egg/cloudinit/netinfo.py", line 447, in debug_info
    netdev_lines = netdev_pformat().splitlines()
  File "/usr/local/lib/python2.7/site-packages/cloud_init-18.3-py2.7.egg/cloudinit/netinfo.py", line 392, in netdev_pformat
    (dev, data["up"], addr["ip"], empty, addr["scope6"],
KeyError: 'scope6'
2018-07-02 11:40:18,204 - util.py[DEBUG]: cloud-init mode 'init' took 0.142 seconds (0.14)
2018-07-02 11:40:18,205 - handlers.py[DEBUG]: finish: init-network: SUCCESS: searching for network datasources

The interface setup on the host is like:

root@host-10-1-80-61:~ # ifconfig -a
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
 ether fa:16:3e:14:1f:99
 hwaddr fa:16:3e:14:1f:99
 inet 10.1.80.61 netmask 0xfffff000 broadcast 10.1.95.255
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet 10Gbase-T <full-duplex>
 status: active
pflog0: flags=0<> metric 0 mtu 33160
pfsync0: flags=0<> metric 0 mtu 1500
 syncpeer: 0.0.0.0 maxupd: 128 defer: off
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 inet6 ::1 prefixlen 128
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
 inet 127.0.0.1 netmask 0xff000000
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

with previous 18.2 release i did not have any problems.

[Ed Fisher (ed-m)]

The regex on scopeid is to blame. If scope is necessary, maybe it's best to determine scope from the ipv6 address itself, instead of from ifconfig output. link for fe80, site for fec0, node for ::1, and global for everything else.

[do3meli (d-info-e)]

meanwhile someone updated the freshport for cloud-init to the version 18.3 with a particular patch that most likely seems to cover this issue: https://svnweb.freebsd.org/ports/head/net/cloud-init/files/patch-cloudinit_netinfo.py?annotate=476024

may someone is able to verify this and make sure it does not create issues with ubuntu or other platforms.

[Mina Galić (minagalic)]

> may someone is able to verify this and make sure it does not create issues with ubuntu or other platforms.

i think you'll be hard pressed finding an Ubuntu or other linux that has no `ip`

[Ryan Harper (raharper)]

Can you capture the 'ip addr show' output from this host?

[Ryan Harper (raharper)]

FreeBSD doesn't have iproute2 (ip) command, it uses the fallback ifconfig output parsing. With that in mind, let's have another look at this.

[Ryan Harper (raharper)]

@do3meli

If you can, please test this branch/change. You can checkout the branch and run it on the target system like so:

% PYTHONPATH=`pwd` python -c 'from cloudinit import netinfo; print(netinfo.netdev_pformat())'

Using the output provided in this bug, on this branch, the output looks like this:

+++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++
+---------+-------+----------------+------------+-------+-------------------+
| Device | Up | Address | Mask | Scope | Hw-Address |
+---------+-------+----------------+------------+-------+-------------------+
| lo0 | True | 127.0.0.1 | 0xff000000 | . | |
| lo0 | True | ::1/128 | . | | |
| lo0 | True | fe80::1%lo0/64 | . | 0x4 | |
| pflog0 | False | . | . | . | |
| pfsync0 | False | . | . | . | |
| vtnet0 | True | 10.1.80.61 | 0xfffff000 | . | fa:16:3e:14:1f:99 |
+---------+-------+----------------+------------+-------+-------------------+

[do3meli (d-info-e)]

@raharper

below a test output that i have just changed and masked some ip address specific information:

+++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++
+---------+-------+-------------------------------------+------------+-------+-------------------+
| Device | Up | Address | Mask | Scope | Hw-Address |
+---------+-------+-------------------------------------+------------+-------+-------------------+
| enc0 | False | . | . | . | . |
| lo0 | True | 127.0.0.1 | 0xff000000 | . | . |
| lo0 | True | 127.0.0.3 | 0xff000000 | . | . |
| lo0 | True | 127.0.0.2 | 0xff000000 | . | . |
| lo0 | True | ::1/128 | . | . | . |
| lo0 | True | fe80::1%lo0/64 | . | 0x5 | . |
| pflog0 | False | . | . | . | . |
| pfsync0 | False | . | . | . | . |
| vtnet0 | True | 10.0.99.33 | 0xfffff000 | . | fa:16:3e:68:3d:ae |
| vtnet1 | True | XXX.XX.51.79 | 0xfffffe00 | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XXX.XX.61.234 | 0xffffffff | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XXX.XX.61.75 | 0xffffffff | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XXX.XX.60.62 | 0xffffffff | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XXX.XX.61.16 | 0xffffffff | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XXX.XX.60.119 | 0xffffffff | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:334f/64 | . | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | fe80::f816:3eff:fef5:30ff%vtnet1/64 | . | 0x2 | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:3dea/128 | . | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:3d4b/128 | . | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:3c3e/128 | . | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:3d10/128 | . | . | fa:16:3e:f5:30:ff |
| vtnet1 | True | XX:XXX:XX:XX:2002:0:YYYY:3c77/128 | . | . | fa:16:3e:f5:30:ff |
| vtnet2 | True | 10.0.122.0 | 0xffff0000 | . | fa:16:3e:e9:7f:a2 |
+---------+-------+-------------------------------------+------------+-------+-------------------+

i will do an additional test on my live environment where deployment of new servers previously failed and report soon.

[do3meli (d-info-e)]

my deployment with cloud-init and FreeBSD 11.2 seems to work with your fix now again:

Cloud-init v. 18.5 finished at Wed, 06 Feb 2019 16:27:26 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 67.8771998882 seconds

[Server Team CI bot (server-team-bot)]

This bug is fixed with commit fff37e7d to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=fff37e7d

[Chad Smith (chad.smith)]

This bug is believed to be fixed in cloud-init in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3204