ssh host keys deleted by cloud-init between sshd-keygen and sshd start
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init |
Invalid
|
Undecided
|
Unassigned |
Bug Description
This happened on a CentOS Stream 8.
I created an AWS instance from a snapshot of another instance.
Upon start I was unable to login via SSH because it failed to start.
Upon log investigation I found out that cloud-init deleted the files from /etc/ssh/ssh_host_* between `sshd-keygen.
I recovered the instance in another way but I dug the logs.
Here are the logs extracts:
messages:
Nov 3 08:30:38 ip-172-21-3-249 systemd[1]: Reached target sshd-keygen.target.
cloud-init.log:
2022-11-03 08:31:02,307 - util.py[DEBUG]: Attempting to remove /etc/ssh/
2022-11-03 08:31:02,308 - util.py[DEBUG]: Attempting to remove /etc/ssh/
2022-11-03 08:31:02,308 - util.py[DEBUG]: Attempting to remove /etc/ssh/
2022-11-03 08:31:02,308 - util.py[DEBUG]: Attempting to remove /etc/ssh/
2022-11-03 08:31:02,308 - util.py[DEBUG]: Attempting to remove /etc/ssh/
2022-11-03 08:31:02,308 - util.py[DEBUG]: Attempting to remove /etc/ssh/
messages:
Nov 3 08:31:02 ip-172-21-3-249 systemd[1]: Starting OpenSSH server daemon...
Nov 3 08:31:03 ip-172-21-3-249 sshd[1337]: Unable to load host key: /etc/ssh/
Nov 3 08:31:03 ip-172-21-3-249 sshd[1337]: Unable to load host key: /etc/ssh/
Nov 3 08:31:03 ip-172-21-3-249 sshd[1337]: Unable to load host key: /etc/ssh/
Nov 3 08:31:03 ip-172-21-3-249 sshd[1337]: sshd: no hostkeys available -- exiting.
Nov 3 08:31:03 ip-172-21-3-249 systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Nov 3 08:31:03 ip-172-21-3-249 systemd[1]: sshd.service: Failed with result 'exit-code'.
Nov 3 08:31:03 ip-172-21-3-249 systemd[1]: Failed to start OpenSSH server daemon.
The cloud-init file has the right dependencies:
[root@ip-
[Unit]
Description=Initial cloud-init job (metadata service crawler)
DefaultDependen
Wants=cloud-
Wants=sshd-
Wants=sshd.service
After=cloud-
After=systemd-
After=network.
After=NetworkMa
Before=
Before=
Before=sshd.service
Before=
[Service]
Type=oneshot
ExecStart=
RemainAfterExit=yes
TimeoutSec=0
# Output needs to appear in instance console output
StandardOutput=
[Install]
WantedBy=
But I wonder if they still work for SystemD templates:
[root@ip-
Unit sshd-keygen.service could not be found.
[root@ip-
Failed to get properties: Unit name sshd-keygen@
[root@ip-
<email address hidden> <email address hidden> <email address hidden> ««« there are 3 services each for it's key type.
I can see that the keygen is disabled here because cloud-init is disabled:
[root@ip-
● <email address hidden> - OpenSSH ed25519 Server Key Generation
Loaded: loaded (/usr/lib/
Drop-In: /etc/systemd/
Active: inactive (dead)
Condition: start condition failed at Thu 2022-11-03 10:18:28 UTC; 3h 4min ago
└─ ConditionPathEx
How can we ensure this does not happen in the future?
[root@ip- 172-21- 3-249 ~]# cloud-init --version
/usr/bin/cloud-init 22.1-5.el8