Comment 8 for bug 1575572

For the record, even after fixing this, installing packages from current cloud-init will still not be 100% reliable in its current form.

The design problem here is that cloud-init runs *in* the boot sequence, i. e. in the transaction where default.target and its dependencies get started. You can't place/start new services into the boot dependency tree (i. e. dependencies of default.target) while default.target is still being started. Any new service which wants to start needs to wait until after the initial boot happens, and will then start afterwards.

invoke-rc.d has some code to deal with that situation, to avoid deadlocks: You can't run the blocking "systemctl start" in a postinst that you run within the boot sequence -- you can at most try to start it without any dependencies and in non-blocking mode, which is why invoke-rc.d has this code:

                # avoid deadlocks during bootup and shutdown from units/hooks
                # which call "invoke-rc.d service reload" and similar, since
                # the synchronous wait plus systemd's normal behaviour of
                # transactionally processing all dependencies first easily
                # causes dependency loops
                if ! OUT=$(systemctl is-system-running 2>/dev/null) && [ "$OUT" != "degraded" ]; then
                    sctl_args="--job-mode=ignore-dependencies"
                fi

This was mostly added for things like /etc/network/ifup.d/ scripts that start stuff, to avoid getting deadlocks on boot. But it would apply here too.

For the most part this should work fine, unless apache expects that any of its dependencies actually get started in that situation (they can't). But if you *do* install a package with an init.d script or service that has dependencies that are not already running, they will *not* be started.

This behaviour of installing packages or configuring your system while the system isn't booted yet might also bite you in other cases (it certainly bit me a number of cases). Maybe we should discuss how to move that after the boot? I do that in https://git.launchpad.net/~ubuntu-release/+git/autopkgtest-cloud/tree/tools/armf-lxd-slave.userdata but this doesn't look very pretty.