Comment 0 for bug 1911680

Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.

We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.

   AuthorizedKeysFile /etc/ssh/authorized_keys

We observed that the file and parent folder permissions had been modified, to:

   /etc/ssh - 0700
   /etc/ssh/authorized_key - 0600

These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.

We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.

https://github.com/canonical/cloud-init/commit/b0e73814db4027dba0b7dc0282e295b7f653325c

While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:

https://github.com/canonical/cloud-init/commit/e161059a18173e2b61c54dba9eab774401fb5f1f

which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.