cloud-init in impish makes /home/ubuntu/.ssh root.root

Bug #1940233 reported by Christian Ehrhardt 
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Critical
Unassigned
cloud-init (Ubuntu)
Critical
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Hirsute
Undecided
Unassigned
Impish
Critical
Unassigned

Bug Description

Hi,
I got to this by my systems complaining to be unable to do ssh-keygen
after deployment. Example:
$ uvt-kvm ssh --insecure impish-kvm 'ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N '\'''\'''
Saving key "/home/ubuntu/.ssh/id_rsa" failed: Permission denied

I found that is due to permissions after guest spawning:
/home/ubuntu/.ssh changed

Old:
drwx------ 2 ubuntu ubuntu 4096 Aug 17 08:20 .ssh/

New:
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

That beaks later things like ssh-keygen.

uvt-kvm only does instruct cloud-init to place a key.
This uses ssh_authorized_keys from
https://cloudinit.readthedocs.io/en/latest/topics/modules.html?highlight=ssh_authorized_keys#authorized-keys

Checked a few guests:
I've seen this on
- impish x86
- impish s390x

I've not seen this on
- bionic
- focal
- impish

You might say - wait a minute impish in both lists.
But it is the date:

Bad
com.ubuntu.cloud.daily:server:21.10:amd64 20210815
cloud-init 21.2-69-g65607405-0ubuntu1

Good
com.ubuntu.cloud.daily:server:21.10:amd64 20210706
cloud-init 21.2-3-g899bfaa9-0ubuntu2

And either this cloud-init version is broken or the underlying new impish image.
I mounted the underlying cloud-image (without customization by cloud-init)
and found that /home is empty (true for all those images).

So to me that seems to be an issue in the new cloud-init that now is in
those images.

Steps to reproduce
# if your host has no keys to push to the guest run ssh-keygen
# sync the latest broken images
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=impish
# spawn guest
$ uvt-kvm create --password=ubuntu i release=impish arch=amd64 label=daily
# wait for it and check the permissions
$ uvt-kvm wait i
$ uvt-kvm ssh i "ls -laF /home/ubuntu/"
drwxr-xr-x 2 root root 4096 Aug 17 08:17 .ssh/

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Collected logs according to https://cloudinit.readthedocs.io/en/latest/topics/bugs.html
(automatic collection seems broken - see 1940235)

summary: - cloud-init in impish makes /home/ubuntu/.ssh root.root breaking its
- use
+ cloud-init in impish makes /home/ubuntu/.ssh root.root
description: updated
Changed in cloud-init (Ubuntu):
importance: Undecided → Critical
tags: added: rls-ii-incoming
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Since it might affect the ability to log into a new spawned system I think this could be release-critical for Impish. Therefore I flagged it like that for now - please feel free to downgrade once the root cause is known and you think a lower rating is appropriate.

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

paride found this change:
https://github.com/canonical/cloud-init/commit/00dbaf1e9ab0e59d81662f0f3561897bef499a3f

It say in the message:
"the directories in the path that do not exist must be root owned and with permission 755."

That is wrong, it needs to be user owned or the user can later on not change things anymore missing write permissions on his own ~/.ssh.

Revision history for this message
Paride Legovini (paride) wrote :

Thanks Christian for filing this.

Yes there's where we introduced the problem. I think we are missing a check in check_create_path() so that directories under the home directory are created as user-owned. Something like:

    newparent_uid = root_pwent.pw_uid
    newparent_gid = root_pwent.pw_gid
    if parent.startswith(home_folder):
        newparent_uid = user_pwent.pw_uid
        newparent_gid = user_pwent.pw_gid

and then:

    util.chownbyid(parent_folder, newparent_uid, newparent_gid)

Maybe we should also check/test how check_create_path() behaves when the user's home directory does not exist, or it is not user-owned and user-readable-and-writable. I'd say that check_create_path() should return False, as it is not this module's duty to create the home directory.

James Falcon (falcojr)
Changed in cloud-init (Ubuntu):
status: New → Triaged
Changed in cloud-init:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Paride Legovini (paride) wrote :

Turns out this bug is also the reason why the HA test jobs are failing, as they run on Impish and and an extra ssh key is added to the instances under test. The failure mode is the same Christian hit.

Revision history for this message
James Falcon (falcojr) wrote :
Revision history for this message
Chad Smith (chad.smith) wrote :

PR landed, expected upload to Ubuntu Impish 21.10 on Monday Aug 23

Changed in cloud-init:
status: Triaged → Fix Committed
Changed in cloud-init (Ubuntu):
status: Triaged → Fix Committed
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thank you all for the prompt handling!

Revision history for this message
James Falcon (falcojr) wrote : Fixed in cloud-init version 21.3.

This bug is believed to be fixed in cloud-init in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 21.3-1-g6803368d-0ubuntu1

---------------
cloud-init (21.3-1-g6803368d-0ubuntu1) impish; urgency=medium

  * New upstream snapshot.
    - testing: Fix ssh keys integration test (#992)
    - Release 21.3 (#993) (LP: #1940839)
    - Azure: During primary nic detection, check interface status continuously
      before rebinding again (#990) [aswinrajamannar]
    - Fix home permissions modified by ssh module (SC-338) (#984)
      (LP: #1940233)
    - Add integration test for sensitive jinja substitution (#986)
    - Ignore hotplug socket when collecting logs (#985) (LP: #1940235)
    - testing: Add missing mocks to test_vmware.py (#982)
    - add Zadara Edge Cloud Platform to the supported clouds list (#963)
      [sarahwzadara]
    - testing: skip upgrade tests on LXD VMs (#980)

 -- James Falcon <email address hidden> Mon, 23 Aug 2021 16:53:23 -0500

Changed in cloud-init (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

While this is important and fixed in the SRU upload of bug 1940871 - the bad code never reached Bionic/Focal/Hirsute as it was in the interim 21.3 versions. Therefore the bug starte for these releases is invalid (it isn't there now, and won't be added in the ongoing SRU).
Setting the states to invalid.

Changed in cloud-init (Ubuntu Bionic):
status: New → Invalid
Changed in cloud-init (Ubuntu Focal):
status: New → Invalid
Changed in cloud-init (Ubuntu Hirsute):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments