cloud-init in impish makes /home/ubuntu/.ssh root.root

[Christian Ehrhardt ]

Hi,
I got to this by my systems complaining to be unable to do ssh-keygen
after deployment. Example:
$ uvt-kvm ssh --insecure impish-kvm 'ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N '\'''\'''
Saving key "/home/ubuntu/.ssh/id_rsa" failed: Permission denied

I found that is due to permissions after guest spawning:
/home/ubuntu/.ssh changed

Old:
drwx------ 2 ubuntu ubuntu 4096 Aug 17 08:20 .ssh/

New:
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

That beaks later things like ssh-keygen.

uvt-kvm only does instruct cloud-init to place a key.
This uses ssh_authorized_keys from
https://cloudinit.readthedocs.io/en/latest/topics/modules.html?highlight=ssh_authorized_keys#authorized-keys

Checked a few guests:
I've seen this on
- impish x86
- impish s390x

I've not seen this on
- bionic
- focal
- impish

You might say - wait a minute impish in both lists.
But it is the date:

Bad
com.ubuntu.cloud.daily:server:21.10:amd64 20210815
cloud-init 21.2-69-g65607405-0ubuntu1

Good
com.ubuntu.cloud.daily:server:21.10:amd64 20210706
cloud-init 21.2-3-g899bfaa9-0ubuntu2

And either this cloud-init version is broken or the underlying new impish image.
I mounted the underlying cloud-image (without customization by cloud-init)
and found that /home is empty (true for all those images).

So to me that seems to be an issue in the new cloud-init that now is in
those images.

Steps to reproduce
# if your host has no keys to push to the guest run ssh-keygen
# sync the latest broken images
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=impish
# spawn guest
$ uvt-kvm create --password=ubuntu i release=impish arch=amd64 label=daily
# wait for it and check the permissions
$ uvt-kvm wait i
$ uvt-kvm ssh i "ls -laF /home/ubuntu/"
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

[Christian Ehrhardt  (paelzer)]

Collected logs according to https://cloudinit.readthedocs.io/en/latest/topics/bugs.html
(automatic collection seems broken - see 1940235)

[Christian Ehrhardt  (paelzer)]

Since it might affect the ability to log into a new spawned system I think this could be release-critical for Impish. Therefore I flagged it like that for now - please feel free to downgrade once the root cause is known and you think a lower rating is appropriate.

[Christian Ehrhardt  (paelzer)]

paride found this change:
https://github.com/canonical/cloud-init/commit/00dbaf1e9ab0e59d81662f0f3561897bef499a3f

It say in the message:
"the directories in the path that do not exist must be root owned and with permission 755."

That is wrong, it needs to be user owned or the user can later on not change things anymore missing write permissions on his own ~/.ssh.

[Paride Legovini (paride)]

Thanks Christian for filing this.

Yes there's where we introduced the problem. I think we are missing a check in check_create_path() so that directories under the home directory are created as user-owned. Something like:

    newparent_uid = root_pwent.pw_uid
    newparent_gid = root_pwent.pw_gid
    if parent.startswith(home_folder):
        newparent_uid = user_pwent.pw_uid
        newparent_gid = user_pwent.pw_gid

and then:

    util.chownbyid(parent_folder, newparent_uid, newparent_gid)

Maybe we should also check/test how check_create_path() behaves when the user's home directory does not exist, or it is not user-owned and user-readable-and-writable. I'd say that check_create_path() should return False, as it is not this module's duty to create the home directory.

[Paride Legovini (paride)]

Turns out this bug is also the reason why the HA test jobs are failing, as they run on Impish and and an extra ssh key is added to the instances under test. The failure mode is the same Christian hit.

[James Falcon (falcojr)]

PR up: https://github.com/canonical/cloud-init/pull/984

[Chad Smith (chad.smith)]

PR landed, expected upload to Ubuntu Impish 21.10 on Monday Aug 23

[Christian Ehrhardt  (paelzer)]

Thank you all for the prompt handling!

[James Falcon (falcojr)]

This bug is believed to be fixed in cloud-init in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 21.3-1-g6803368d-0ubuntu1

---------------
cloud-init (21.3-1-g6803368d-0ubuntu1) impish; urgency=medium

  * New upstream snapshot.
    - testing: Fix ssh keys integration test (#992)
    - Release 21.3 (#993) (LP: #1940839)
    - Azure: During primary nic detection, check interface status continuously
      before rebinding again (#990) [aswinrajamannar]
    - Fix home permissions modified by ssh module (SC-338) (#984)
      (LP: #1940233)
    - Add integration test for sensitive jinja substitution (#986)
    - Ignore hotplug socket when collecting logs (#985) (LP: #1940235)
    - testing: Add missing mocks to test_vmware.py (#982)
    - add Zadara Edge Cloud Platform to the supported clouds list (#963)
      [sarahwzadara]
    - testing: skip upgrade tests on LXD VMs (#980)

 -- James Falcon <email address hidden> Mon, 23 Aug 2021 16:53:23 -0500

[Christian Ehrhardt  (paelzer)]

While this is important and fixed in the SRU upload of bug 1940871 - the bad code never reached Bionic/Focal/Hirsute as it was in the interim 21.3 versions. Therefore the bug starte for these releases is invalid (it isn't there now, and won't be added in the ongoing SRU).
Setting the states to invalid.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3894