start hook fails after a reboot: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 8404: invalid start byte

[Nobuto Murata]

Once a host is rebooted after installing hwe-edge kernel (v5.8) on focal, the charm status is stuck at "hook failed: start".

unit-ovn-chassis-14: 06:16:40 INFO unit.ovn-chassis/14.juju-log CompletedProcess(args=('update-alternatives', '--set', 'ovs-vswitchd', '/usr/lib/openvswitch-switch/ovs-vswitchd'), returncode=0, stdout='')
unit-ovn-chassis-14: 06:16:40 INFO unit.ovn-chassis/14.juju-log Invoking reactive handler: reactive/ovn_chassis_charm_handlers.py:75:configure_bridges
unit-ovn-chassis-14: 06:16:40 ERROR unit.ovn-chassis/14.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ovn-chassis-14/charm/reactive/ovn_chassis_charm_handlers.py", line 79, in configure_bridges
    charm_instance.configure_bridges()
  File "lib/charms/ovn_charm.py", line 515, in configure_bridges
    bpi = os_context.BridgePortInterfaceMap(bridges_key=self.bridges_key)
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/context.py", line 2725, in __init__
    for ifname in list_nics():
  File "/var/lib/juju/agents/unit-ovn-chassis-14/.venv/lib/python3.8/site-packages/charmhelpers/core/host.py", line 844, in list_nics
    ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 8404: invalid start byte

It looks like charm-helper is using `ip a` and I see some non-ASCII characters in the output in this system especially in "altname".

278: enp129s0f0np0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:0b:03:e1:48:5e brd ff:ff:ff:ff:ff:ff
    altname prb<F6>vU
    altname <D0>?<DE>/<C0>^?
22: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 36:41:e1:bd:6c:4d brd ff:ff:ff:ff:ff:ff
279: eth127: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:ad:e7:dd:14:7c brd ff:ff:ff:ff:ff:ff
    altname 0<93>b<F6>vU

[Nobuto Murata (nobuto)]

Subscribing ~field-critical.

[Nobuto Murata (nobuto)]

exported bundle:
https://pastebin.canonical.com/p/FvtBVS44D8/

[Frode Nordahl (fnordahl)]

The root of the issue is that the virtual ports created by the card contain garbage in the `altname` field:

279: eth127: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 06:ad:e7:dd:14:7c brd ff:ff:ff:ff:ff:ff
    altname 0�b�vU

[Nobuto Murata (nobuto)]

The full output of `ip a`.
https://pastebin.canonical.com/p/FmwtJ2FSXS/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/763513

[Frode Nordahl (fnordahl)]

I have made a hotfix charm available that contains these changes in cs:~fnordahl/ovn-chassis-bug1904978

Note that since it contains a unreleased version of charm-helpers you need to manually remove the charm virtualenv prior to upgrading the charm, otherwise the charm-helpers code in the virtualenv will not be updated.

[Nobuto Murata (nobuto)]

https://github.com/juju/charm-helpers/pull/533

[Frode Nordahl (fnordahl)]

https://github.com/juju/charm-helpers/pull/533

[Aurelien Lourot (aurelien-lourot)]

backport to 20.10: https://review.opendev.org/c/x/charm-ovn-chassis/+/763723