Unable to connect with openssh 7.8 client and certificates

[Scott Emmons]

Users are unable to connect to Ubuntu when using openssh client 7.8 and certificates. We have seen this with both xenial and bionic, but this affects connecting to ANY host running openssh server <7.8.

It appears to be specific to using certificate authentication.

The only known recourse at this time is either downgrade clients to 7.7 or a previous version of openssh, or create new keys/certificates with a different alg that is acceptable for both the older server and newer client.

The error message via ssh -vvv is:
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:REDACTED
debug1: send_pubkey_test: no mutual signature algorithm

When comparing the list returned from a 7.6 server and a 7.8 server via "ssh -Q key", we find that 7.8 returns <email address hidden> and <email address hidden> which are not present (or valid) for the earlier version server.

It appears that the change noted here in the release notes[1] for 7.8 is related:
 * sshd(8): the semantics of PubkeyAcceptedKeyTypes and the similar
   HostbasedAcceptedKeyTypes options have changed. These now specify
   signature algorithms that are accepted for their respective
   authentication mechanism, where previously they specified accepted
   key types. This distinction matters when using the RSA/SHA2
   signature algorithms "rsa-sha2-256", "rsa-sha2-512" and their
   certificate counterparts. Configurations that override these
   options but omit these algorithm names may cause unexpected
   authentication failures (no action is required for configurations
   that accept the default for these options).

This is also affecting other Linux distributions as well:
https://bugzilla.redhat.com/show_bug.cgi?id=1623929
https://bugs.archlinux.org/task/59838

[1] https://www.openssh.com/txt/release-7.8

[Jakub (jakub-redhat-bugs)]

Description of problem:
The OpenSSH server in RHEL7.6 does not send complete list of signature algorithms in SHA2 extension.

debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>

This causes failures if the client is on OpenSSH 7.8p1+ (Fedora 28+) and for some reason disabled the rsa-sha2-* public key algorithms with PubkeyAcceptedKeyTypes configuration option.

The correct list should look like this:

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,null>

This does not affect any other key types at this moment.

Version-Release number of selected component (if applicable):
openssh-7.4p1-16

How reproducible:
specific configuration

Steps to Reproduce:
1. Install OpenSSH 7.8p1 (Fedora 28+)
2. Configure pubkey authentication using RSA key with remote server example.com
3. ssh -vvv -o PubkeyAcceptedKeyTypes=ssh-rsa example.com

Actual results:

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:I1XXiJ/wkXC6Vn8ohZVHcJTCCKoPKm4mL8qtjtyNMhw /home/lslebodn/.ssh/id_rsa
debug1: send_pubkey_test: no mutual signature algorithm

Expected results:

The authentication should proceed using ssh-rsa algorithm.

Additional info:
This is a change in OpenSSH 7.8 that it is getting more strict about handling this extension. Unfortunately we carry broken version in RHEL7, which is not sending complete list of algorithms.

Workaround:
In client, list also the SHA2 extension algorithms:

PubkeyAcceptedKeyTypes rsa-sha2-256,rsa-sha2-512

If you need to adjust this list, rather use the + sign.

Thanks lslebodn for reporting this issue to me.

[Christoph (christoph-redhat-bugs)]

Hi,

client fedora 28 with openssh-7.8p1-2.fc28.x86_64

using a <email address hidden> client certificate

> debug1: Offering public key: RSA-CERT > SHA256:xxx /home/c/.ssh/id_rsa-cert.pub
> debug1: send_pubkey_test: no mutual signature algorithm

The proposed workaround does not seem to work (Even if adding the cert type)

> PubkeyAcceptedKeyTypes rsa-sha2-256,rsa-sha2-512,ssh-rsa,<email address hidden>

[Jakub (jakub-redhat-bugs)]

If you want to use certificates, you need to list also the SHA2 variants of certificates:

<email address hidden>,<email address hidden>

Not sure if this is somewhere documented, but is should do the job.

[Christoph (christoph-redhat-bugs)]

Hi,

i tried

PubkeyAcceptedKeyTypes rsa-sha2-256,rsa-sha2-512,<email address hidden>,<email address hidden>,<email address hidden>

but still

debug1: send_pubkey_test: no mutual signature algorithm

[Jakub (jakub-redhat-bugs)]

Please, open a customer case if you have this issue with your RHEL installation.

https://access.redhat.com/

This will really need a fix in RHEL7 since the new OpenSSH checks the signature algorithms against the hardcoded list there, which is wrong.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Emmons (lscotte)]

Updated the description - specifically, this appears to affect certificate authentication and be related to <email address hidden> and <email address hidden> which are present in 7.8 server, but not earlier versions (nor are valid to add to the configuration manually).

[Etienne (etienne-redhat-bugs)]

Hi all,

Even if the CA is an RSA key, you can sign ECDSA or ED25519 keys so you get ECDSA/ED25519 certs which allow you to work around the issue without changing anything server-side

Exemple cert:
$ ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub
~/.ssh/id_ed25519-cert.pub:
        Type: <email address hidden> user certificate
        Public key: ED25519-CERT SHA256:<...>
        Signing CA: RSA SHA256:<...>
        Key ID: "..."

[Christian Ehrhardt  (paelzer)]

Thanks Scott for the report, we don't have 7.8 yet (Cosmic will be released with 7.7).
But as I read the issue this already is an issue for all older SSH servers as soon as somebodies client is new.

@cjwatson - you are usually watching over openssh, did you see this already. How should we proceeed in your opinion.

[Etienne CHAMPETIER (champtar)]

Hi all,

This only affect RSA certificates, not ECDSA or ED25519 certs
Even if your CA is an RSA key, you can sign ECDSA or ED25519 public keys so you get ECDSA/ED25519 certificates which allow you to work around the issue without changing anything server-side
(and without deploying a new CA)

Exemple of working cert (7.8 client, <7.8 server):
$ ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub
~/.ssh/id_ed25519-cert.pub:
        Type: <email address hidden> user certificate
        Public key: ED25519-CERT SHA256:<...>
        Signing CA: RSA SHA256:<...>
        Key ID: "..."

[Scott Emmons (lscotte)]

This [1] appears to be the source of the problem, specifically "Add new RSA certificate types that that can be used in the above options and on the wire to require the use of RSA/SHA2 signatures." - unfortunately, those new certificate types don't exist/work in openssh <7.8, breaking backwards compatibility with 7.8 clients.

Christian - Correct, it doesn't matter that no Ubuntu version is shipping with openssh 7.8 today. Bleeding edge distributions are, and non-Linux users are getting updates to 7.8, which breaks connectivity to any openssh server <7.8 under these circumstances when the client is 7.8.

Etienne - Thank you for providing that - it is the current workaround aside from downgrading clients to 7.7. This is not a complete solution though, as it doesn't help for environments that sign RSA user certificates through an automated service (unless that service supports EC certs, which I'm going to guess may not work with really old versions of openssh).

[1] http://bugzilla.mindrot.org/show_bug.cgi?id=2799

[Christoph (christoph-redhat-bugs)]

I think something happend within openssh 7.9 if my interpretation of this is correct: https://www.spinics.net/lists/openssh-unix-dev/msg05371.html

[Jakub (jakub-redhat-bugs)]

Indeed, there is fix [1] in latest OpenSSH 7.9p1 so updating the clients to the latest version should resolve the issue. But it does not change that there is a bug in RHEL7 too.

[1] https://github.com/openssh/openssh-portable/commit/1a4a9cf8

[Robie Basak (racb)]

Am I right in understanding that there is no proposed action for Ubuntu to take on this right now?

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:7.9p1-1

---------------
openssh (1:7.9p1-1) unstable; urgency=medium

  * New upstream release (https://www.openssh.com/txt/release-7.9):
    - ssh(1), sshd(8): allow most port numbers to be specified using service
      names from getservbyname(3) (typically /etc/services; closes:
      #177406).
    - ssh(1): allow the IdentityAgent configuration directive to accept
      environment variable names. This supports the use of multiple agent
      sockets without needing to use fixed paths.
    - sshd(8): support signalling sessions via the SSH protocol. A limited
      subset of signals is supported and only for login or command sessions
      (i.e. not subsystems) that were not subject to a forced command via
      authorized_keys or sshd_config.
    - ssh(1): support "ssh -Q sig" to list supported signature options.
      Also "ssh -Q help" to show the full set of supported queries.
    - ssh(1), sshd(8): add a CASignatureAlgorithms option for the client and
      server configs to allow control over which signature formats are
      allowed for CAs to sign certificates. For example, this allows
      banning CAs that sign certificates using the RSA-SHA1 signature
      algorithm.
    - sshd(8), ssh-keygen(1): allow key revocation lists (KRLs) to revoke
      keys specified by SHA256 hash.
    - ssh-keygen(1): allow creation of key revocation lists directly from
      base64-encoded SHA256 fingerprints. This supports revoking keys using
      only the information contained in sshd(8) authentication log messages.
    - ssh(1), ssh-keygen(1): avoid spurious "invalid format" errors when
      attempting to load PEM private keys while using an incorrect
      passphrase.
    - sshd(8): when a channel closed message is received from a client,
      close the stderr file descriptor at the same time stdout is closed.
      This avoids stuck processes if they were waiting for stderr to close
      and were insensitive to stdin/out closing (closes: #844494).
    - ssh(1): allow ForwardX11Timeout=0 to disable the untrusted X11
      forwarding timeout and support X11 forwarding indefinitely.
      Previously the behaviour of ForwardX11Timeout=0 was undefined.
    - sshd(8): when compiled with GSSAPI support, cache supported method
      OIDs regardless of whether GSSAPI authentication is enabled in the
      main section of sshd_config. This avoids sandbox violations if GSSAPI
      authentication was later enabled in a Match block.
    - sshd(8): do not fail closed when configured with a text key revocation
      list that contains a too-short key.
    - ssh(1): treat connections with ProxyJump specified the same as ones
      with a ProxyCommand set with regards to hostname canonicalisation
      (i.e. don't try to canonicalise the hostname unless
      CanonicalizeHostname is set to 'always').
    - ssh(1): fix regression in OpenSSH 7.8 that could prevent public-key
      authentication using certificates hosted in a ssh-agent(1) or against
      sshd(8) from OpenSSH <7.8 (LP: #1790963).
    - All: support building against the openssl-1.1 API (releases 1.1.0g and
      later). The openssl-1.0 AP...