Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKeysFile /etc/ssh/authorized_keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700
/etc/ssh/authorized_key - 0600
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2-g371b392c-0ubuntu1~16.04.1 to 20.4-0ubuntu1~16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/authorized_keys, then writes the key for the disabled root user to the same location, overwriting the previous write.
Starting on the 6th January 2021 we started observing SSH authentication issues in AWS AMI builds.
We have SSH configured with an absolute (i.e. rather than per-user) authorised keys file, e.g.
AuthorizedKe ysFile /etc/ssh/ authorized_ keys
We observed that the file and parent folder permissions had been modified, to:
/etc/ssh - 0700 ssh/authorized_ key - 0600
/etc/
These permissions would be fine if the authorised keys file were in a users home directory, but not for a centrally owned absolute file.
We investigated and identified that between the 4th and 6th January 2021, cloud-init on Ubuntu 16.04 was upgraded from 20.3-2- g371b392c- 0ubuntu1~ 16.04.1 to 20.4-0ubuntu1~ 16.04.1. The newer version included the following fix, that led to the problem, although prior commits set the scene for this commit to cause us trouble.
https:/ /github. com/canonical/ cloud-init/ commit/ b0e73814db4027d ba0b7dc0282e295 b7f653325c
While trying workarounds (e.g. oneshot service to revert permissions), we then ran into another change that appended exit(142) to the command option:
https:/ /github. com/canonical/ cloud-init/ commit/ e161059a18173e2 b61c54dba9eab77 4401fb5f1f
which then meant, as root is disabled, that SSH would not work using the authorised key pair for any user. This is because cloud-init first writes the key for the user (e.g. ubuntu) and in our case writing the key to /etc/ssh/ authorized_ keys, then writes the key for the disabled root user to the same location, overwriting the previous write.