juju breaks cloud-init cloud-config

[Brett Holman]

Problem
=======

Juju's docs attempt to document the issue:

> Specifying a key will overwrite what juju puts in the cloudinit file with the following caveats:
>
> 1. users and bootcmd keys will cause an error
> 2. The packages key will be appended to the packages listed by juju
> 3. The `runcmds` key will cause an error. You can specify `preruncmd` and `postruncmd` keys to prepend and append the runcmd created by Juju.

While this is a good starting point, it falls short.

The cloudinit-userdata user interface[1] is confusing in that it attempts to allow a user to use user-data cloud-config, however it forces the user to use a modified format that cloud-init does not directly support. In the case of `{pre,post}runcmd`, it presents the user with two alternatives without giving them the information required to choose which. Why would a user prefer preruncmd over postruncmd and vice versa? I.e. what does juju inject into `runcmd` that the user should be thinking about when making this decision? I was going to suggest documenting this better, but after looking at the code I came to the conclusion that this is not a simple thing to document. Additionally, I couldn't come up with a case where a user would prefer preruncmd[2] (perhaps I missed something).

At a minimum we should give the user the information they need to make this decision, however this might be an opportunity to improve Juju's UI, which is why I'm filing here rather than on juju's docs discourse.

Proposal
========

A simpler user interface might be to automatically intercept runcmd and insert the user’s runcmd after juju’s runcmd data, like it does with adding user packages to the end of juju-created ones. This would allow the user to not have to think about the syntax differences between juju-flavored user-data and the version[3] that is portable across all clouds using cloud-init directly.

Having different syntax and semantics for juju's user-data leads to bugreports to cloud-init containing keys that cloud-init has never supported (which we now document[4]), bugreports [5], and general user confusion related to why there are so many `#cloud-config` formats and why they are all different.

There is a lot to consider in this issue. As a starting point for better compatibility with cloud-init, I propose:

1) fix postruncmd semantic differences[5]

2) add support for cloud-init's `runcmd` syntax and semantics
   - support `runcmd`: make a new key `runcmd` behave the way that postruncmd does (without the set -ex side-effect)
   - possibly deprecate preruncmd?
   - possibly deprecate postruncmd?

3) add support for `bootcmd`

[1] https://juju.is/docs/juju/list-of-model-configuration-keys#heading--cloudinit-userdata
[2] outside of the `set -e` bug[6] which changes the expected cloud-init runcmd behavior
[3] https://cloudinit.readthedocs.io/en/latest/reference/modules.html
[4] https://cloudinit.readthedocs.io/en/latest/reference/faq.html#autoinstall-preruncmd-postruncmd
[5] https://bugs.launchpad.net/juju/+bug/1978454
[6] https://github.com/juju/juju/blob/42f6a4bbb7818847c3fd3234d39da98369c4a03f/cloudconfig/userdatacfg_unix.go#L175

[Brett Holman (holmanb)]

I just filed a PR[1] to address the first item in the proposal.

[1] https://github.com/juju/juju/pull/17180

[Harry Pidcock (hpidcock)]

Thanks for the bug report and PR.