cloud-init.service/start deleted to break ordering cycle

[Rossen]

I'm encountering this systemd service dependecy cycle, which prevent cloud-init from running, and making my image build unusable since user-data settings do not get applied.

Issues present on bionic and focal

Steps to reproduce:

1. apt-get purge -y ufw; apt-get install -y firewalld
2. cloud-init clean
3. reboot
4. grep -B 5 cycle /var/log/syslog

I've seen these two error messages:

```
[ 3.609277] systemd[1]: Set hostname to <localhost>.
[ 3.620361] systemd[1]: Initializing machine ID from KVM UUID.
[ 3.621151] systemd[1]: Installed transient /etc/machine-id file.
[ 4.007739] systemd[1]: basic.target: Found ordering cycle on paths.target/start
[ 4.017890] systemd[1]: basic.target: Found dependency on apport-autoreport.path/start
[ 4.018837] systemd[1]: basic.target: Found dependency on sysinit.target/start
[ 4.019713] systemd[1]: basic.target: Found dependency on cloud-init.service/start
[ 4.020613] systemd[1]: basic.target: Found dependency on systemd-networkd-wait-online.service/start
[ 4.021700] systemd[1]: basic.target: Found dependency on systemd-networkd.service/start
[ 4.022677] systemd[1]: basic.target: Found dependency on network-pre.target/start
[ 4.023574] systemd[1]: basic.target: Found dependency on firewalld.service/start
[ 4.024463] systemd[1]: basic.target: Found dependency on basic.target/start
[ 4.025298] systemd[1]: basic.target: Job paths.target/start deleted to break ordering cycle starting with basic.target/start
[[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;1;39mPaths[0m
[ 4.027404] systemd[1]: firewalld.service: Found ordering cycle on sysinit.target/start
[ 4.028367] systemd[1]: firewalld.service: Found dependency on cloud-init.service/start
[ 4.029359] systemd[1]: firewalld.service: Found dependency on systemd-networkd-wait-online.service/start
[ 4.034786] systemd[1]: firewalld.service: Found dependency on systemd-networkd.service/start
[ 4.035843] systemd[1]: firewalld.service: Found dependency on network-pre.target/start
[ 4.036853] systemd[1]: firewalld.service: Found dependency on firewalld.service/start
[ 4.037878] systemd[1]: firewalld.service: Job cloud-init.service/start deleted to break ordering cycle starting with firewalld.service/start
[[0;1;31m SKIP [0m] Ordering cycle found, skipping [0;���job (metadata service crawler)[0m
[ 4.041631] systemd[1]: Created slice system-modprobe.slice.
```

```
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.824604] systemd[1]: sysinit.target: Found ordering cycle on cloud-init.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.825678] systemd[1]: sysinit.target: Found dependency on systemd-networkd-wait-online.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.826718] systemd[1]: sysinit.target: Found dependency on systemd-networkd.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.827613] systemd[1]: sysinit.target: Found dependency on network-pre.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.828450] systemd[1]: sysinit.target: Found dependency on firewalld.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.836078] systemd[1]: sysinit.target: Found dependency on dbus.service/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.836928] systemd[1]: sysinit.target: Found dependency on basic.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.837814] systemd[1]: sysinit.target: Found dependency on paths.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.838762] systemd[1]: sysinit.target: Found dependency on ua-license-check.path/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.839750] systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Jan 6 22:04:02 ib-scan-ubuntu2004 kernel: [ 3.840612] systemd[1]: sysinit.target: Job cloud-init.service/start deleted to break ordering cycle starting with sysinit.target/start
```

[Chad Smith (chad.smith)]

Thank you for filing a bug and making Ubuntu better.

I can confirm that your procedure to reproduce on Ubuntu jammy. This does in fact represent a bug in how the firewalld systemd unit is being setup. My first guess here is the DefaultDependencies being injected by firewalld's systemd service is what is causing the ordering cycle w/ cloud-init.
It will need a bit more time to triage. But I can at least confirm the bug report.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chad Smith (chad.smith)]

This looks to also have an existing issue over in firewalld too https://github.com/firewalld/firewalld/issues/414

[Ross Vandegrift (ross-kallisti)]

The Debian cloud team received #[1025618](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025618), which is a report of this bug.

I tried to follow the suggestion at https://github.com/firewalld/firewalld/issues/414#issuecomment-1061607756 but it's not quite right. It suggests dropping `Before=network-pre.target` from `cloud-init-local.service`. But:
- `cloud-init-local.service` has `Before=network-pre.target`, but is not involved in the cycles. The cycle details posted above only involve `cloud-init.service`.
   - the debian bug has a nice python script to find cycles in `systemd-analyze dot --order` output.
- `cloud-init.service` has no explicit order with `network-pre.target`, but it does have `Before=sysinit.target`.

The closest I could get to Michael's suggestion is dropping `Before=sysinit.target` from `cloud-init.service` - but this will break cloud-init's disk setup features (see cloud-init rev `80f5ec4b` which introduces `Before=basic.target`).

Could the disk and network setup pieces of cloud init be separated? That'd allow keeping disk setup before `sysinit.target`, but permitting network setup to move later (though I suspect that brings risks too, LP#1636912 in particular?)