| 2021-01-14 12:09:36 |
Kevin Rogers |
bug |
|
|
added bug |
| 2021-01-14 12:10:31 |
Kevin Rogers |
description |
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write. |
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.
There are similarities to https://bugs.launchpad.net/cloud-init/+bug/1839061, but this is a different issue. |
|
| 2021-01-14 14:46:17 |
Dan Watkins |
cloud-init: assignee |
|
Dan Watkins (oddbloke) |
|
| 2021-01-14 14:46:20 |
Dan Watkins |
cloud-init: status |
New |
In Progress |
|
| 2021-01-14 14:46:22 |
Dan Watkins |
cloud-init: importance |
Undecided |
High |
|
| 2021-01-14 17:39:20 |
Dan Watkins |
information type |
Public |
Private Security |
|
| 2021-01-18 19:12:16 |
Dan Watkins |
attachment added |
|
Verification logs for groovy LXD https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454233/+files/groovy.lxd-container.log |
|
| 2021-01-18 19:15:23 |
Dan Watkins |
attachment added |
|
Verification logs for groovy EC2 https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454234/+files/groovy.ec2.log |
|
| 2021-01-18 19:16:30 |
Dan Watkins |
attachment added |
|
Verification logs for focal Azure https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454235/+files/focal.azure.log |
|
| 2021-01-18 19:17:24 |
Dan Watkins |
attachment added |
|
Verification logs for focal EC2 https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454236/+files/focal.ec2.log |
|
| 2021-01-18 19:27:53 |
Dan Watkins |
description |
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.
There are similarities to https://bugs.launchpad.net/cloud-init/+bug/1839061, but this is a different issue. |
[Impact]
For users who provide more than one unique SSH key to cloud-init and have a shared AuthorizedKeysFile configured in sshd_config, cloud-init 20.4 started writing all of these keys to such a file, granting all such keys SSH access as root. Previously, cloud-init did not handle such sshd_config configuration, and so would have written users' keys to the hardcoded `~/.ssh/authorized_keys` unconditionally.
[Test Case]
This test case has been written for use with the cloud-init integration testing framework:
```
ABSOLUTE_AUTH_KEYS_PATH = "/etc/ssh/authorized_keys"
BACKDOOR_KEYS = "/etc/backdoor_keys"
USER_DATA_TMPL = """\
#cloud-config
bootcmd:
- sed -i "s,#AuthorizedKeysFile.*,AuthorizedKeysFile {0} {1}," /etc/ssh/sshd_config
runcmd:
- |
cat << EOF > {1}
{{}}
EOF
""".format(ABSOLUTE_AUTH_KEYS_PATH, BACKDOOR_KEYS)
# This test checks that we do not write keys into an absolute
# AuthorizedKeysFile by default, because doing so presents a security issue.
#
# Because we're intentionally testing that we do not write authorized keys to
# the location configured for sshd, we cannot SSH into the system using keys in
# that path. We use runcmd to populate a second configured path to grant us
# access; runcmd will install the key after all the SSH key determination has
# been completed, therefore not affecting it.
def test_test(session_cloud, setup_image):
# We shouldn't write out absolute files until we've figured out how to do
# it right
user_data = USER_DATA_TMPL.format(
session_cloud.cloud_instance.key_pair.public_key_content
)
with session_cloud.launch(user_data=user_data) as client:
assert client.execute(
"test -f {}".format(ABSOLUTE_AUTH_KEYS_PATH)
).failed
```
[Regression Potential]
This update touches only cloud-init's SSH handling, by reverting a commit which modified its behaviour. It has been present in Ubuntu images since the 6th of January.
Users who have started relying on the new behaviour (cloud-init writing keys to non-standard AuthorizedKeysFile locations configured in sshd_config) in the past ~2 weeks will see this behaviour stop working.
[Original Report]
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.
There are similarities to https://bugs.launchpad.net/cloud-init/+bug/1839061, but this is a different issue. |
|
| 2021-01-18 19:34:03 |
Dan Watkins |
attachment added |
|
Verification logs for bionic LXD containers https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454238/+files/bionic.lxd-container.log |
|
| 2021-01-18 19:58:44 |
Dan Watkins |
attachment added |
|
Verification logs for groovy GCE https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454251/+files/groovy.gce.log |
|
| 2021-01-18 20:01:32 |
Dan Watkins |
bug |
|
|
added subscriber Brian Murray |
| 2021-01-18 20:01:42 |
Dan Watkins |
bug |
|
|
added subscriber Łukasz Zemczak |
| 2021-01-18 20:14:24 |
Dan Watkins |
attachment added |
|
Verification logs for groovy Azure https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454252/+files/groovy.azure.log |
|
| 2021-01-18 20:18:36 |
Dan Watkins |
attachment added |
|
Verification logs for bionic EC2 https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454253/+files/bionic.ec2.log |
|
| 2021-01-18 20:19:15 |
Dan Watkins |
attachment added |
|
Verification logs for xenial Azure https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454254/+files/xenial.azure.log |
|
| 2021-01-18 20:22:13 |
Dan Watkins |
attachment added |
|
Verification logs for xenial GCE https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454255/+files/xenial.gce.log |
|
| 2021-01-18 20:23:47 |
Dan Watkins |
attachment added |
|
Verification logs for focal LXD containers https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454256/+files/focal.lxd-container.log |
|
| 2021-01-18 20:27:57 |
Dan Watkins |
attachment added |
|
Verification logs for focal GCE https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454257/+files/focal.gce.log |
|
| 2021-01-18 20:29:02 |
Dan Watkins |
attachment added |
|
Verification logs for bionic Azure https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454258/+files/bionic.azure.log |
|
| 2021-01-18 21:05:40 |
Dan Watkins |
attachment added |
|
Verification logs for bionic GCE https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454268/+files/bionic.gce.log |
|
| 2021-01-18 21:38:49 |
Dan Watkins |
attachment added |
|
Bug verification logs for LXD https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454269/+files/lxd-container.single |
|
| 2021-01-18 21:39:21 |
Dan Watkins |
attachment added |
|
Bug verification logs for Azure https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454270/+files/azure.single |
|
| 2021-01-18 21:39:51 |
Dan Watkins |
attachment added |
|
Bug verification logs for EC2 https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454271/+files/ec2.single |
|
| 2021-01-18 21:40:12 |
Dan Watkins |
attachment added |
|
Bug verification logs for GCE https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454272/+files/gce.single |
|
| 2021-01-18 21:58:13 |
Dan Watkins |
attachment added |
|
Verification logs for xenial LXD containers https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454273/+files/xenial.lxd-container.log |
|
| 2021-01-18 22:47:05 |
Dan Watkins |
attachment added |
|
Verification logs for groovy LXD VMs https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454279/+files/groovy.lxd-vm.log |
|
| 2021-01-18 23:29:45 |
Dan Watkins |
attachment added |
|
Verification logs for focal LXD VMs https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454306/+files/focal.lxd-vm.log |
|
| 2021-01-18 23:30:35 |
Dan Watkins |
attachment added |
|
Verification logs for bionic LXD VMs https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454307/+files/bionic.lxd-vm.log |
|
| 2021-01-19 00:00:38 |
Dan Watkins |
attachment added |
|
Verification logs for xenial LXD VMs https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454308/+files/xenial.lxd-vm.log |
|
| 2021-01-19 00:11:56 |
Dan Watkins |
attachment added |
|
Bug verification logs for LXD VMs https://bugs.launchpad.net/cloud-init/+bug/1911680/+attachment/5454309/+files/lxd-vm.single |
|
| 2021-01-19 16:06:06 |
Dan Watkins |
information type |
Private Security |
Public Security |
|
| 2021-07-05 21:13:30 |
James Falcon |
cloud-init: status |
In Progress |
Fix Released |
|
| 2023-05-12 11:21:39 |
James Falcon |
bug watch added |
|
https://github.com/canonical/cloud-init/issues/3829 |
|
