SSH-RSA not supported for Self-SSH in Ubuntu 22.04 FIPS

[Arunaav Alok]

On a FIPS Enabled Ubuntu 22.04 kernel, we are seeing an issue with self-ssh.
We created a key with the following steps:
touch /home/core/.ssh/known_hosts
  ssh-keygen -q -t rsa -f /home/core/.ssh/id_rsa -N '' > /dev/null
  cp /home/core/.ssh/id_rsa.pub /home/core/.ssh/authorized_keys
  chmod 0600 /home/core/.ssh/id_rsa
  chmod 0600 /home/core/.ssh/authorized_keys

When we try to do a self ssh with the key, the following happens:
ssh -i .ssh/id_rsa onprem_shell@10.14.169.25
Connection closed by 10.14.169.25 port 22

FIPS status:
cat /proc/sys/crypto/fips_enabled
1

PFB, the ssh dump:

ssh -v user@10.14.169.25
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: FIPS mode initialized
debug1: Connecting to 10.14.169.25 [10.14.169.25] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1
debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.14.169.25:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-cbc MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes128-cbc MAC: hmac-sha1 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 10.14.169.25 port 22

hostname -i
10.14.169.25

Please note that SSH onto other hosts (both FIPS and non-FIPS) works. The only workaround that we have found has been removing the ssh-rsa entry from “HostKeyAlgorithms” in “etc/ssh/sshd_config” and restarting the SSH service. This issue has neither been encountered in the Ubuntu 18.04 FIPS nor Ubuntu 20.04 FIPS.

[João Gomes (gomesjoao)]

Hi Arunaav, thank you for your report.
How did you install the FIPS packages?
Can you also provide the server log with debug enabled?

[Seth Arnold (seth-arnold)]

Hello Arunaav, I'm curious if you could double-check the testing environment to make sure the user accounts are as you expected?

chmod 0600 /home/core/.ssh/authorized_keys
ssh -i .ssh/id_rsa onprem_shell@10.14.169.25
ssh -v user@10.14.169.25
debug1: identity file /root/.ssh/id_rsa type -1

There's usernames 'core', 'onprem_shell', 'user', and 'root' in play here, and I think it'd be extraordinarily easy to perhaps use sudo or another privilege changing tool in such a way that it is using the wrong private key or the wrong authorized_keys file, etc.

Thanks

[Arunaav Alok (arnv-alok)]

Hi João, we are installing FIPS packages by using the Pro token.

PFB, the detailed debug logs of the SSH connection

ssh -vv onprem_shell@10.14.169.25
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 10.14.169.25 is address
debug1: FIPS mode initialized
debug1: Connecting to 10.14.169.25 [10.14.169.25] port 22.
debug1: Connection established.
debug1: identity file /home/core/.ssh/id_rsa type -1
debug1: identity file /home/core/.ssh/id_rsa-cert type -1
debug1: identity file /home/core/.ssh/id_ecdsa type -1
debug1: identity file /home/core/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/core/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/core/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/core/.ssh/id_ed25519 type -1
debug1: identity file /home/core/.ssh/id_ed25519-cert type -1
debug1: identity file /home/core/.ssh/id_ed25519_sk type -1
debug1: identity file /home/core/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/core/.ssh/id_xmss type -1
debug1: identity file /home/core/.ssh/id_xmss-cert type -1
debug1: identity file /home/core/.ssh/id_dsa type -1
debug1: identity file /home/core/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1
debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6+Fips1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.14.169.25:22 as 'onprem_shell'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,ext-info-c,<email address hidden>
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug2: ciphers ctos: aes128-cbc,aes256-cbc,aes128-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: ciphers stoc: aes128-cbc,aes256-cbc,aes128-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: MACs ctos: hmac-sha1
debug2: MACs stoc: hmac-sha1
debug2: compression ctos: none,<email address hidden>,zlib
debug2: compression stoc: none,<email address hidden>,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,<email address hidden>
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256
debug2: ciphers ctos: aes128-cbc,aes256-cbc,aes128-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: ciphers stoc: aes128-cbc,aes256-cbc,aes128-ctr,aes256-ctr,<email address hidden>,<email address hidden>
debug2: MACs ctos: hmac-sha1
debug2: MACs stoc: hmac-sha1
debug2: compression ctos: none,<email address hidden>
debug2: compression stoc: none,<email address hidden>
debug2: languages cto...

[Arunaav Alok (arnv-alok)]

Hi Seth,

I do not think that we are using the wrong authorized key or keyfiles here, I have double checked. Also, if we were using the wrong keys, we would see the same issue in Ubuntu 18.04 and 20.04 (FIPS and non-FIPS), however, this is not so. We are only seeing this issue with Ubuntu 22.04 FIPS.

Thanks