Hello,
I am running this ami "ami-003ab7b6dd427580e" on aws ec2.
It is the rhel 8 quickstart ami and an arm image running 4.18.0-240.8.1.el8_3.aarch64
I run it on a m6g.xlarge machine and it has cloud-init pre-installed.
Sadly, it isn't printing the ssh host key keys on first boot:
```
[ OK ] Reached target Login Prompts.
[FAILED] Failed to start OpenSSH server daemon.
See 'systemctl status sshd.service' for details.
[ 115.600243] cloud-init[1097]: Cloud-init v. 19.4 running 'modules:config' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.52 seconds.
[ OK ] Started Apply the settings specified in cloud-config.
Starting Execute cloud user/final scripts...
[ OK ] Started System Logging Service.
[ OK ] Reached target Multi-User System.
Starting Update UTMP about System Runlevel Changes...
[ OK ] Started Update UTMP about System Runlevel Changes.
ci-info: +++++++++++++++++++++Authorized keys from /home/ec2-user/.ssh/authorized_keys for user ec2-user++++++++++++++++++++++
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | - | packer_5ff2df88-947a-5d37-29ff-4ca186ea3f79 |
ci-info: | ssh-rsa | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | - | jenkins |
ci-info: +---------+-------------------------------------------------+---------+---------------------------------------------+
<14>Jan 4 15:36:36 ec2:
<14>Jan 4 15:36:36 ec2: #############################################################
<14>Jan 4 15:36:36 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Jan 4 15:36:36 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Jan 4 15:36:36 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----
[ 116.110779] cloud-init[1180]: Cloud-init v. 19.4 running 'modules:final' at Mon, 04 Jan 2021 15:36:36 +0000. Up 115.90 seconds.
[ 116.112793] cloud-init[1180]: Cloud-init v. 19.4 finished at Mon, 04 Jan 2021 15:36:36 +0000. Datasource DataSourceEc2Local. Up 116.10 seconds
[ 116.190100] echo[1376]: trying to reload or restart NetworkManager.service
[ OK ] Started Execute cloud user/final scripts.
[ OK ] Reached target Cloud-init target.
```
ssh also failes to come up at first, but succeeds on second try like this:
```
sshd.service log:
Jan 04 13:49:15 xxx systemd[1]: Starting OpenSSH server daemon...
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_rsa_key
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 04 13:49:15 xxx sshd[1092]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
Jan 04 13:49:15 xxx sshd[1092]: sshd: no hostkeys available -- exiting.
Jan 04 13:49:15 xxx systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 13:49:15 xxx systemd[1]: sshd.service: Failed with result 'exit-code'.
Jan 04 13:49:15 xxx systemd[1]: Failed to start OpenSSH server daemon.
Jan 04 13:49:57 xxx systemd[1]: sshd.service: Service RestartSec=42s expired, scheduling restart.
Jan 04 13:49:57 xxx systemd[1]: sshd.service: Scheduled restart job, restart counter is at 1.
Jan 04 13:49:57 xxx systemd[1]: Stopped OpenSSH server daemon.
Jan 04 13:49:58 xxx systemd[1]: Starting OpenSSH server daemon...
Jan 04 13:49:58 xxx sshd[4720]: Server listening on 0.0.0.0 port 22.
Jan 04 13:49:58 xxx sshd[4720]: Server listening on :: port 22.
Jan 04 13:49:58 xxx systemd[1]: Started OpenSSH server daemon.
Jan 04 13:50:45 xxx sshd[4722]: Accepted publickey for ec2-user from xxx port 55550 ssh2: RSA xxx
Jan 04 13:50:45 xxx sshd[4722]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
```
The /etc/ssh directory also is populated:
```
-rw-r--r--. 1 root root 577388 Mar 27 2020 moduli
-rw-r--r--. 1 root root 1770 Mar 27 2020 ssh_config
drwxr-xr-x. 2 root root 28 Nov 26 08:58 ssh_config.d
-rw-r-----. 1 root ssh_keys 480 Jan 4 17:18 ssh_host_ecdsa_key
-rw-r--r--. 1 root root 162 Jan 4 17:18 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys 387 Jan 4 17:18 ssh_host_ed25519_key
-rw-r--r--. 1 root root 82 Jan 4 17:18 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys 2578 Jan 4 17:18 ssh_host_rsa_key
-rw-r--r--. 1 root root 554 Jan 4 17:18 ssh_host_rsa_key.pub
-rw-------. 1 root root 4268 Jan 4 09:28 sshd_config
```
Hi Karsten, thanks for filing this bug! It's not really clear to me what's going on here: cloud-init should be writing out SSH host keys during cc_ssh execution, which we can see happens in the log.
My best guess as to what is going on is that there is either (a) some `ssh_keys` configuration which doesn't specify any valid keys: this would cause us to take the path which doesn't _generate_ the keys, but also wouldn't write out any keys; or, (b) there is an empty `ssh_genkeytypes` configuration which means that we take the generation path, but have no key types to generate. By looking at the journal, I can see that sshd-keygen service instances are running: this could mean that (a) or (b) are intentionally configured in the base image you're building from (because they expect sshd-keygen to perform this, so don't want cloud-init to do so). (It could also mean that sshd-keygen is actively removing the keys that cloud-init created; I don't know what the service does, exactly, so I can't be sure.)
I can see from the logs that no user-data is specified to the instance: this means that any such configuration would have to be in /etc/cloud somewhere.
Could you pastebin the contents of /etc/cloud/
Thanks!
Dan