/etc/ssh/ssh_host_rsa_key fails to be generated when the sshd service Is restarted after the cloud-Init boot Is completed

[shixuantong]

It should be related to the following commit:
https://github.com/canonical/cloud-init/commit/b3e31ba228d32c318872fb68edda272f679e1004

[shixuantong (sxt1001)]

Can I delete this file (/run/systemd/generator.early/multi-user.target.wants/cloud-init.target) after the cloud-init boot is finished?

[shixuantong (sxt1001)]

/etc/ssh/ssh_host_rsa_key fails to be generated:

[root@localhost ~]# cloud-init status
status: done
[root@localhost ~]# systemctl status cloud-init
○ cloud-init.service - Initial cloud-init job (metadata service crawler)
     Loaded: loaded (/usr/lib/systemd/system/cloud-init.service; enabled; vendor preset: disabled)
     Active: inactive (dead)

Feb 04 15:20:21 localhost cloud-init[1909]: ci-info: ... ...
... ...
[root@localhost ~]# rm -rf /etc/ssh/ssh_host_rsa_key
[root@localhost ~]# systemctl restart sshd
[root@localhost ~]# ll /etc/ssh/ssh_host_rsa_key
ls: cannot access '/etc/ssh/ssh_host_rsa_key': No such file or directory
[root@localhost ~]#

[shixuantong (sxt1001)]

No matter whether cloud-init is inactive or active, /etc/ssh/ssh_host_rsa_key generation is affected.

[Emanuele Esposito (esposem)]

If you use cloud-init, then a drop-in disable-sshd-keygen-if-cloud-init-active.conf will be placed to prevent ssh creating keys and race with cloud-init.
If you then disable cloud-init, but the drop-in is still there, you won't have any process taking care of ssh keys because cloud-init is disabled and sshd is stopped because of the drop-in.

[shixuantong (sxt1001)]

but when cloud-init has finished running, sshd is always stopped because of disable-sshd-keygen-if-cloud-init-active.conf. Is that not a problem?

[Alberto Contreras (aciba)]

Hello, shixuantong.

cloud-init will create the ssh keys using the cc_ssh module on first boot, and sshd-keygen won't run.

Could you please elaborate on where you think the problem is, and/or what your expectations are?

[shixuantong (sxt1001)]

Hello, Alberto Contreras, thank you very much for your reply.

I have such a use scenario here. After the cloud-init boot is complete, ssh_host_rsa_key is lost due to certain reasons (for example, manual deletion). I need to restart the sshd service to generate ssh_host_rsa_key. This scenario cannot be executed successfully because of disable-sshd-keygen-if-cloud-init-active.conf.

[Huijuan Zhao (huijuanzhao)]

Hello shixuantong,

Could you please help to check if the below workaround works for you?

Generate ssh host keys manually via the below commands:
/usr/libexec/openssh/sshd-keygen rsa
/usr/libexec/openssh/sshd-keygen ecdsa
/usr/libexec/openssh/sshd-keygen ed25519

Then restart sshd service

Thanks!

[Alberto Contreras (aciba)]

Then you could either

1) Manually regenerate the key(s) as Huijuan Zhao suggested or
2) Remove the systemd/disable-sshd-keygen-if-cloud-init-active.conf file and restart the service or the system.

(2) is probably safe, after the first boot has run, as `cc_ssh` only runs on first boot. Thus, subsequent boots won't race with sshd-keygen.
Additionally, in your use case, you are generating a new set of keys which you are going to manually extract, so it does not matter if the keys are generated by cloud-init or sshd-keygen.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4070