write_files runs before users/groups, renders "owner" useless

[Graham Leggett]

When the following cloud-init script is run the expectation is that a group called ssl-cert-client is created, and this group is applied to the file that is written via the "owner" tag.

groups:
  - ssl-cert-server
  - ssl-cert-client
write_files:
- encoding: gzip
  content: !!binary |
$(echo ${rsa_client_private_key} | gzip - | openssl base64 | sed -e "s/^/ /")
  owner: root:ssl-cert-client
  path: /etc/ssl/certs/${resourcegroup}-${machine}.${domain}-client.key
  permissions: '0640'

What happens instead is that the writing of the file is attempted before the creation of the group, and this file write fails because the group ssl-cert-server does not yet exist.

The two tasks need to be swapped round before they are practically useful.

[Joshua Powers (powersj)]

Thank you for taking the time to report this bug. In an effort to keep
an up-to-date and valid list of bugs to work on, I have reviewed this
report verifying it still requires effort and occurs on a supported
version of Ubuntu.

I do agree that the situation you spell out occurs. Consider the
following simplier example:

userdata.yaml:
#cloud-config
groups:
  - testgroup
write_files:
- encoding: file
  content: test data
  owner: root:testgroup
  path: /root/testfile
  permissions: '0640'

This should write a file to /root/testfile with the permissions
set to root:testgroup.

$ lxc init ubuntu-daily:x x
$ lxc config set x user.user-data - < userdata.yaml
$ lxc start x
$ lxc exec x2 -- ls -l /root/testfile
-rw-r----- 1 root root 9 Jul 27 21:42 /root/testfile

[Scott Moser (smoser)]

To be fair, this isn't completely useless.
If we simply moved the writing of files to after users were created, then less of the boot could be affected by your written files.

As a (possibly contrived) example, as it is right now you can use write_files to replace 'adduser' and then cloud-init would call the replaced version of adduser when adding users. If write_files happened later you could not do that.

I would like to get this to work though.
2 options:
a.) add 'write_files_late' that runs after users are created (or write_files_early and write_files) basically 2 different points in boot.
b.) magically determine if 'owner' isn't around that we should re-try this after the usergroups are created.
c.) extend format of write_files to include 'early' or 'late' that would indicate to cloud-init explicitly that it should happen after users or before users creation.

[Pedro A. Aranda (paaguti-p)]

I accept your reasoning. And following it, I would add the write_files_late for “my” purpose ;-) and leave the current write_files as is. Either this, or include a new sub item in users, like users - <user> - write_files to write files of a specific user...

Don’t know how you feel about this alternative. But it seems logical and would be backwards-compatible

Best, /PA

Enviado desde mi iPad

> El 13 jul 2018, a las 18:17, Scott Moser <email address hidden> escribió:
>
> To be fair, this isn't completely useless.
> If we simply moved the writing of files to after users were created, then less of the boot could be affected by your written files.
>
> As a (possibly contrived) example, as it is right now you can use
> write_files to replace 'adduser' and then cloud-init would call the
> replaced version of adduser when adding users. If write_files happened
> later you could not do that.
>
> I would like to get this to work though.
> 2 options:
> a.) add 'write_files_late' that runs after users are created (or write_files_early and write_files) basically 2 different points in boot.
> b.) magically determine if 'owner' isn't around that we should re-try this after the usergroups are created.
> c.) extend format of write_files to include 'early' or 'late' that would indicate to cloud-init explicitly that it should happen after users or before users creation.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1781549).
> https://bugs.launchpad.net/bugs/1486113
>
> Title:
> write_files runs before users/groups, renders "owner" useless
>
> Status in cloud-init:
> Confirmed
>
> Bug description:
> When the following cloud-init script is run the expectation is that a
> group called ssl-cert-client is created, and this group is applied to
> the file that is written via the "owner" tag.
>
> groups:
> - ssl-cert-server
> - ssl-cert-client
> write_files:
> - encoding: gzip
> content: !!binary |
> $(echo ${rsa_client_private_key} | gzip - | openssl base64 | sed -e "s/^/ /")
> owner: root:ssl-cert-client
> path: /etc/ssl/certs/${resourcegroup}-${machine}.${domain}-client.key
> permissions: '0640'
>
> What happens instead is that the writing of the file is attempted
> before the creation of the group, and this file write fails because
> the group ssl-cert-server does not yet exist.
>
> The two tasks need to be swapped round before they are practically
> useful.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1486113/+subscriptions

[Adam Spiers (adam.spiers)]

This was first reported in 2013 in bug #1231541 and I still see the issue :-/ Is there any reason not to simply just swap the order? Why would it ever be a problem to run write_files after users_groups?

[Scott Moser (smoser)]

On Thu, Jan 10, 2019, 1:30 PM Adam Spiers <<email address hidden> wrote:

> This was first reported in 2013 in bug #1231541 and I still see the
> issue :-/

I understand the frustration.

Is there any reason not to simply just swap the order?

Backwards compatibility

  Why
> would it ever be a problem to run write_files after users_groups?
>

The simple example of writing a file to /etc/skel that you wanted to have
copied into created users' do.

>

[manuel (boillodmanuel)]

Hello,

I wonder why there is a backwards compatibility issue.

As write_files runs before users/groups, the current version of write_files should use existing users or group. If not, the write_files should fail.

If we swap the order to runs write_files after users/groups, the existing write_files should still works, don't they?

[Dan Watkins (oddbloke)]

On Wed, Apr 15, 2020 at 05:01:12PM -0000, manuel wrote:
> I wonder why there is a backwards compatibility issue.
>
> As write_files runs before users/groups, the current version of
> write_files should use existing users or group. If not, the write_files
> should fail.
>
> If we swap the order to runs write_files after users/groups, the
> existing write_files should still works, don't they?

The example Scott gave would not generate a traceback, but it would fail
in the sense that the created users would not have the contents that
were added to /etc/skel. The proposed change in order would lead to a
regression for users who are using this (or similar) configurations.

[manuel (boillodmanuel)]

Hi Dan,

Thanks for the quick reply. I read carefully the Scott answer, and I got it :) It's somehow weird, but I understand that it can break compatibility.

I guess 95% of people will benefits from creating users before write_files, but it can break something for other 5%.

So, it's an 5y old issue, and I see only 2 exits:
- add a breaking change (I don't now what is your policy about that)
- add this feature without breaking change with one of the solutions Scott proposed (except the second one 😀 )

[Ian T Price (iantprice)]

I've just hit this seven year old issue and wasted a good morning trying to figure it out - many thanks to @Odd_Bloke on FreeNode #cloud-init for steering me here.

Scott's Option 2a is the obvious answer here to add the functionality and still maintain backwards compatability.

Plus 1 for 2a

[Brian Candler (b-candler)]

It's not just write_files. It appears that runcmd also runs before users/groups, so you can't do this as a workaround:

runcmd:
  - |
    cat <<EOS >/home/foo/bar
    ...
    EOS
  - chown foo:foo /home/foo/bar

My comments:

1. It would be helpful if the order in which modules are run were documented.

2. I'd like users/groups to be created early - controlled by a flag if backwards-compatibility is required. Otherwise: write_files, runcmd etc are unable to depend on the existence of a uid/gid.

The only downside I can see is that extra files dropped in /etc/skel wouldn't be honoured - but these could always be be created explicitly using write_files.

[Dan Watkins (oddbloke)]

On Tue, Apr 28, 2020 at 09:02:20AM -0000, Brian Candler wrote:
> It's not just write_files. It appears that runcmd also runs before
> users/groups, so you can't do this as a workaround:

This is not the case in cloud-init's default configuration.

> 1. It would be helpful if the order in which modules are run were
> documented.

The order of modules is configured in /etc/cloud/cloud.cfg, so you can
see exactly the order your image is configured to use there. The
template that is used to generate cloud.cfg upstream is
https://github.com/canonical/cloud-init/blob/master/config/cloud.cfg.tmpl

(I agree that we should surface this info in our documentation!)

> 2. I'd like users/groups to be created early - controlled by a flag if
> backwards-compatibility is required. Otherwise: write_files, runcmd etc
> are unable to depend on the existence of a uid/gid.

On an Ubuntu system (which uses the upstream order), these are the first
set of modules executed:

    cloud_init_modules:
     - migrator
     - seed_random
     - bootcmd
     - write-files
     - growpart
     - resizefs
     - disk_setup
     - mounts
     - set_hostname
     - update_hostname
     - update_etc_hosts
     - ca-certs
     - rsyslog
     - users-groups
     - ssh

As you can see, write-files is the only one that is likely to be
user/group-sensitive. (runcmd is in the next stanza, executed later in
boot.)

If your image is configured this way and you're still seeing an issue
using runcmd, please file a separate bug. :)

[Henry Ford (hrford)]

Hi,

Regarding comment from manuel (boillodmanuel): I feel I'm one of the 95% of users where write_files would be very useful to occur after the users module.

Currently, my work-around is:
runcmd:
  - cloud-init single --name write_files --frequency once

...which runs over the same files created before, but now applies the correct permissions, at least on an EC2 with Amazon Linux 2.

My question is, as this bug is still "open" is there an official work-around?

[Dan Watkins (oddbloke)]

The workaround is to use runcmd (which runs later in boot) to either (a) write the files in their entirety, or (b) fix up the owners/permissions/locations of the files. (Using it to rerun the entire module is good if it works for you, though I'd want to spend more time thinking about potential consequences before recommending it myself. :)

The exact shape of the workaround depends on what you're doing, really: if you have large base64 encoded files then using `write_files` to get them onto disk and then `runcmd` to shift them around probably makes sense; if you're writing out a single small file, then just doing it in `runcmd` probably makes most sense.

If anyone is interested in contributing a doc fix to mention this ordering issue in the `write_files` docs (https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_write_files.py#L44-L60) and/or a more general fix to introduce a `write_files` stage executed later in boot (as suggested by Scott in comment #2), both would be most welcome! (In the latter case, though, please do reach out so we can determine a specific path forward!)

[Lucen Dio (lucendio)]

Hi everyone,

I created https://github.com/canonical/cloud-init/pull/916 to start solutioning. I tried to summarize the different approaches and I'd love to get your feedback. Maybe we can continue the conversation over there. It would also be the chance to share other approaches that come to your mind!

[Brett Holman (holmanb)]

This was resolved in the aforementioned pull request with the addition of writing deferred files.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2573