debian 10 cloud-init stuck in systemd-networkd after 1. reboot

[do3meli]

i have create a virtual machine using the following official debian image: https://cdimage.debian.org/cdimage/openstack/current-10/

they have version 18.3-6 installed.

after the initial boot of the vm cloud-init works as it should. but when a reboot occurred it seems to be hanging in networkd-systemd bringing up interfaces. see attached log file.

i have also manually upgraded to cloud-init 19.2-1 and that had the same issue.

[Dan Watkins (oddbloke)]

Thanks for the bug report! Are you able to run `cloud-init collect-logs` on the affected system and attach the output to this bug? (Please set this back to New if either you can't, or once you have. :)

[Chad Smith (chad.smith)]

Thank you for filing this bug against cloud-init. Is it possible to run sudo cloud-init collect-logs and attach the emitted tar.gz to this bug

[Dan Watkins (oddbloke)]

Hi do3meli, am I right in thinking that you are using systemd-networkd but have ifupdown installed? Could you see if the problem persists if ifupdown isn't installed?

[do3meli (d-info-e)]

Hi Dan,

the mentioned out of the box debian 10 image does indeed have ifupdown installed. after the initial boot i see ifupdown tools available and systemd-networkd is available too, but the service is disabled. unfortunately removing ifupdown wont realy help as it would remove cloud-init too:

root@xxxx:/home/debian# apt purge ifupdown
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  cloud-init* ifenslave* ifupdown*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 2094 kB disk space will be freed.

note this is on a freshly installed vm with the following image: https://cdimage.debian.org/cdimage/openstack/current-10/

any idea what's going wrong with:

a) the boot process
b) the package dependencies?

[Dan Watkins (oddbloke)]

Aha, OK, the initial report is a little misleading. You _aren't_ using systemd-networkd for networking, as far as I can tell, as `ifup` is involved in the networking configuration. Could you paste the output of `find /etc/network`, please?

[do3meli (d-info-e)]

sure, here we go:

root@xxxx:/home/debian# find /etc/network
/etc/network
/etc/network/interfaces.d
/etc/network/interfaces.d/50-cloud-init.cfg
/etc/network/interfaces
/etc/network/if-post-down.d
/etc/network/if-post-down.d/ifenslave
/etc/network/if-down.d
/etc/network/if-up.d
/etc/network/if-up.d/ifenslave
/etc/network/if-pre-up.d
/etc/network/if-pre-up.d/ifenslave

[Ryan Harper (raharper)]

The journal.log file indicates that your dhclient is not happy with the DHCP server and keeps trying to get a valid lease.

Oct 03 13:40:20.243288 doxtestbox sh[457]: DHCPACK of 217.26.49.116 from 217.26.49.123
Oct 03 13:40:20.243193 doxtestbox dhclient[515]: DHCPACK of 217.26.49.116 from 217.26.49.123
Oct 03 13:40:20.245248 doxtestbox sh[457]: RTNETLINK answers: File exists
Oct 03 13:40:20.262567 doxtestbox root[533]: /etc/dhcp/dhclient-exit-hooks.d/rfc3442-classless-routes returned non-zero exit status 2
Oct 03 13:40:20.270702 doxtestbox dhclient[515]: bound to 217.26.49.116 -- renewal in 36799 seconds.
Oct 03 13:40:20.270997 doxtestbox sh[457]: bound to 217.26.49.116 -- renewal in 36799 seconds.
Oct 03 13:40:22.128459 doxtestbox dhclient[590]: Internet Systems Consortium DHCP Client 4.4.1
Oct 03 13:40:22.128955 doxtestbox sh[457]: Internet Systems Consortium DHCP Client 4.4.1
Oct 03 13:40:22.128955 doxtestbox sh[457]: Copyright 2004-2018 Internet Systems Consortium.
Oct 03 13:40:22.128955 doxtestbox sh[457]: All rights reserved.
Oct 03 13:40:22.128955 doxtestbox sh[457]: For info, please visit https://www.isc.org/software/dhcp/
Oct 03 13:40:22.128476 doxtestbox dhclient[590]: Copyright 2004-2018 Internet Systems Consortium.
Oct 03 13:40:22.128480 doxtestbox dhclient[590]: All rights reserved.
Oct 03 13:40:22.128484 doxtestbox dhclient[590]: For info, please visit https://www.isc.org/software/dhcp/
Oct 03 13:40:22.128488 doxtestbox dhclient[590]:
Oct 03 13:40:22.135170 doxtestbox dhclient[590]: Listening on Socket/eth0
Oct 03 13:40:22.135454 doxtestbox sh[457]: Listening on Socket/eth0
Oct 03 13:40:22.135454 doxtestbox sh[457]: Sending on Socket/eth0
Oct 03 13:40:22.135454 doxtestbox sh[457]: PRC: Soliciting for leases (INIT).
Oct 03 13:40:22.135205 doxtestbox dhclient[590]: Sending on Socket/eth0
Oct 03 13:40:23.006336 doxtestbox sh[457]: XMT: Forming Solicit, 0 ms elapsed.
Oct 03 13:40:23.006336 doxtestbox sh[457]: XMT: X-- IA_NA 3e:73:d8:19
Oct 03 13:40:23.006336 doxtestbox sh[457]: XMT: | X-- Request renew in +3600
Oct 03 13:40:23.006336 doxtestbox sh[457]: XMT: | X-- Request rebind in +5400
Oct 03 13:40:23.006856 doxtestbox sh[457]: XMT: Solicit on eth0, interval 1080ms.
Oct 03 13:40:23.006399 doxtestbox dhclient[590]: XMT: Solicit on eth0, interval 1080ms.
Oct 03 13:40:23.007768 doxtestbox dhclient[590]: RCV: Advertise message on eth0 from fe80::f816:3eff:fecb:2aec.

[Ryan Harper (raharper)]

I suspect you can debug this by logging in via serial console and then test with:

ifup --debug eth0

and possibly adding set +x to any of the ifup hook scripts and the dhclient hook scripts.

[Ryan Harper (raharper)]

I'm marking this invalid for cloud-init as the logs do not indicate there is any cloud-init specific failure. As you debug if you find something that cloud-init should have done, please move this bug state back to New.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3457