juju breaks cloud-init cloud-config

[Brett Holman]

Problem
=======

Juju's docs attempt to document the issue:

> Specifying a key will overwrite what juju puts in the cloudinit file with the following caveats:
>
> 1. users and bootcmd keys will cause an error
> 2. The packages key will be appended to the packages listed by juju
> 3. The `runcmds` key will cause an error. You can specify `preruncmd` and `postruncmd` keys to prepend and append the runcmd created by Juju.

While this is a good starting point, it falls short.

The cloudinit-userdata user interface[1] is confusing in that it attempts to allow a user to use user-data cloud-config, however it forces the user to use a modified format that cloud-init does not directly support. In the case of `{pre,post}runcmd`, it presents the user with two alternatives without giving them the information required to choose which. Why would a user prefer preruncmd over postruncmd and vice versa? I.e. what does juju inject into `runcmd` that the user should be thinking about when making this decision? I was going to suggest documenting this better, but after looking at the code I came to the conclusion that this is not a simple thing to document. Additionally, I couldn't come up with a case where a user would prefer preruncmd[2] (perhaps I missed something).

At a minimum we should give the user the information they need to make this decision, however this might be an opportunity to improve Juju's UI, which is why I'm filing here rather than on juju's docs discourse.

Proposal
========

A simpler user interface might be to automatically intercept runcmd and insert the user’s runcmd after juju’s runcmd data, like it does with adding user packages to the end of juju-created ones. This would allow the user to not have to think about the syntax differences between juju-flavored user-data and the version[3] that is portable across all clouds using cloud-init directly.

Having different syntax and semantics for juju's user-data leads to bugreports to cloud-init containing keys that cloud-init has never supported (which we now document[4]), bugreports [5], and general user confusion related to why there are so many `#cloud-config` formats and why they are all different.

There is a lot to consider in this issue. As a starting point for better compatibility with cloud-init, I propose:

1) fix postruncmd semantic differences[5]

2) add support for cloud-init's `runcmd` syntax and semantics
   - support `runcmd`: make a new key `runcmd` behave the way that postruncmd does (without the set -ex side-effect)
   - possibly deprecate preruncmd?
   - possibly deprecate postruncmd?

3) add support for `bootcmd`

[1] https://juju.is/docs/juju/list-of-model-configuration-keys#heading--cloudinit-userdata
[2] outside of the `set -e` bug[6] which changes the expected cloud-init runcmd behavior
[3] https://cloudinit.readthedocs.io/en/latest/reference/modules.html
[4] https://cloudinit.readthedocs.io/en/latest/reference/faq.html#autoinstall-preruncmd-postruncmd
[5] https://bugs.launchpad.net/juju/+bug/1978454
[6] https://github.com/juju/juju/blob/42f6a4bbb7818847c3fd3234d39da98369c4a03f/cloudconfig/userdatacfg_unix.go#L175

[Brett Holman (holmanb)]

I just filed a PR[1] to address the first item in the proposal.

[1] https://github.com/juju/juju/pull/17180

[Harry Pidcock (hpidcock)]

Thanks for the bug report and PR.

[Brett Holman (holmanb)]

From a discussion with the Juju team, it seems that perhaps cloud-config isn't supported in a generic sense, but rather Juju and allows a few specific cloud-config keys to be set by the user. This is at odds with the current docs, which say:

> The cloudinit-userdata allows the user to provide additional cloudinit data to be included in the cloudinit data created by Juju.
>
> Specifying a key will overwrite what juju puts in the cloudinit file with the following caveat

From the conversation we had, it sounds like the Juju team will want to decide whether we want to try to fully support cloud-config.

If so, the content in this original bug report is still relevant. If the preference is to only allow a few keys, then instead of what I described above I think we will want to rework the docs (and possibly userdata) to more correctly describe what works rather than implying (as it currently does) that all of cloud-config[1] is currently supported with just a few caveats.

[1] https://docs.cloud-init.io/en/latest/explanation/format.html#cloud-config-data