Bionic: system update killed networking (sort of)

[Tim Gardner]

Filing for a friend:

I have several Ubuntu servers that all just quit working at the same time. By “quit working” I mean they take forever to boot and applications do not work on them correctly until after about 30 minutes (and the applications have been restarted).

Nov 14 16:27:08 help systemd[1]: Started OpenBSD Secure Shell server.
Nov 14 16:27:08 help whoopsie[775]: [16:27:08] Using lock path: /var/lock/whoopsie/lock
Nov 14 16:27:09 help whoopsie[775]: [16:27:09] Could not get the Network Manager state:
Nov 14 16:27:09 help whoopsie[775]: [16:27:09] GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.NetworkManager has not provided by any .service files
Nov 14 16:27:09 help networkd-dispatcher[684]: WARNING: systemd-networkd is not running, output will be incomplete.
Nov 14 16:27:09 help networkd-dispatcher[684]: ERROR:Unknown state for interface NetworkctlListState(idx=1, name='lo', type='loopback', operational='n/a', administrative='unmanaged'): n/a
Nov 14 16:27:09 help networkd-dispatcher[684]: Traceback (most recent call last):
Nov 14 16:27:09 help networkd-dispatcher[684]: File "/usr/bin/networkd-dispatcher", line 286, in trigger_all
Nov 14 16:27:09 help networkd-dispatcher[684]: force=True)
Nov 14 16:27:09 help networkd-dispatcher[684]: File "/usr/bin/networkd-dispatcher", line 338, in handle_state
Nov 14 16:27:09 help networkd-dispatcher[684]: raise UnknownState(operational_state)
Nov 14 16:27:09 help networkd-dispatcher[684]: UnknownState: n/a
Nov 14 16:27:09 help networkd-dispatcher[684]: ERROR:Unknown state for interface NetworkctlListState(idx=2, name='eth0', type='ether', operational='n/a', administrative='unmanaged'): n/a
Nov 14 16:27:09 help networkd-dispatcher[684]: Traceback (most recent call last):
Nov 14 16:27:09 help networkd-dispatcher[684]: File "/usr/bin/networkd-dispatcher", line 286, in trigger_all
Nov 14 16:27:09 help networkd-dispatcher[684]: force=True)
Nov 14 16:27:09 help networkd-dispatcher[684]: File "/usr/bin/networkd-dispatcher", line 338, in handle_state
Nov 14 16:27:09 help networkd-dispatcher[684]: raise UnknownState(operational_state)
Nov 14 16:27:09 help networkd-dispatcher[684]: UnknownState: n/a
Nov 14 16:27:09 help systemd[1]: Started Dispatcher daemon for systemd-networkd.

[Tim Gardner (timg-tpi)]

After restoring to a functional backup we discovered that updating from cloud-init 22.2-0ubuntu1~18.04.3 is the cause of the problem.

[Chad Smith (chad.smith)]

Thank you for filing this bug for a friend and making cloud-init better. So looks like an upgrade path affected these systems on Bionic. If possible do we have the ability to run `sudo cloud-init collect-logs` on an affected upgraded system and attach the resulting tarfile?

What we'd like to know to better triage this issue:
  - Was there custom network config provided in either /etc/netplan or /etc/NetworkManager
  - nmcli general
  - netplan get all
  - what version of systemd was on the affected system
  - the contents of /run/systemd/network files
  - the cloud-ini.tar.gz from cloud-init collect logs (which gives us journal info, logs etc)
      NOTICE: if `sudo cloud-init query userdata` on the affected system contains secrets or passwords you'll want to avoid providing userdata at the collect-logs prompt

[Chad Smith (chad.smith)]

I've set this bug status to 'Incomplete' above to indicate cloud-init devs are waiting for additional content/logs communication on this bug to be able to better triage the problem here.

If you get a chance on an affected node to grab the additional logs, please set this back to 'New' state so we can be alerted that updates are available. Thanks again

[Launchpad Janitor (janitor)]

[Expired for cloud-init (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for cloud-init (Ubuntu Bionic) because there has been no activity for 60 days.]