Issue on sshd finds correct private key for a certificate when using ssh-agent

Bug #1952421 reported by Utkarsh Gupta
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openssh (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Chloé Smith
Hirsute
Fix Released
Undecided
Chloé Smith
Impish
Fix Released
Undecided
Utkarsh Gupta

Bug Description

Reported as https://bugzilla.mindrot.org/show_bug.cgi?id=3254 upstream

[Impact]

 * HostCertificate and HostKeyAgent are not working together in sshd due
   to a mismatched certificate's public key and private key. The function ` `sshkey_equal_public()`` incorrectly compares the certificate's public
  key with a private key, never finding a match. The impact is that sshd
  cannot use said certificate *even though* its private key is indeed in
  ssh-agent.

* What it should do is compare the certificate's public key with a public key in `sensitive_data`.

* Having this SRU-ed is a direct ask from one of the major cloud partners.
  They are currently using a customised version of the package to work
  around this issue, and we would like them to use a package directly from
  our own archive.

 * Looping through sensitive_data.host_pubkeys[j] *instead* of
   sensitive_data.host_keys[j] fixes the issue

[https://github.com/openssh/openssh-portable/blob/V_8_4/sshd.c#L1936]

/* Find matching private key */
 for (j = 0; j < options.num_host_key_files; j++) {
  if (sshkey_equal_public(key,
   sensitive_data.host_keys[j])) {
   sensitive_data.host_certificates[j] = key;
    break;
   }
 }

vs.

/* Find matching private key */
 for (j = 0; j < options.num_host_key_files; j++) {
  if (sshkey_equal_public(key,
   sensitive_data.host_pubkeys[j])) {
   sensitive_data.host_certificates[j] = key;
    break;
   }
 }

[Test Plan]

 * Due to the empirical nature of this bug, the test is quite straight
   forward. *Without* the fix, one cannot use certificates to authenticate
   successfully (e.g. ``sshd -c /path/to/certificate.pem``)
   whereas with the fix (assuming the certificate matches a host key) you
   can create a channel.

[Where problems could occur]

 * This has already been fixed both upstream and in Jammy without issue.
   However, if a regression where to happen it would probably be in one of
   two ways:

     * A dependency/reverse-dependency issue stemming from the version
       bump that will happen if this fix is ported. We mitigate this risk
       by testing for these exact types of regression,
       and by selecting carefully what to label this new version.

     * Accidentally breaking a set up that was made to work around this
       bug in the first place. The risk of this is lower, as the most
       likely fix is the one being implemented here anyway. Though
       to mitigate this more we can describe exactly what is happening
       with the fix in the changelog.

This affects every version of openssh back until Focal, at least.

Related branches

Utkarsh Gupta (utkarsh)
Changed in openssh (Ubuntu):
status: New → Fix Released
Chloé Smith (kajiya)
description: updated
Utkarsh Gupta (utkarsh)
Changed in openssh (Ubuntu Impish):
assignee: nobody → Utkarsh Gupta (utkarsh)
Changed in openssh (Ubuntu Hirsute):
assignee: nobody → Chloé Smith (kajiya)
Changed in openssh (Ubuntu Focal):
assignee: nobody → Chloé Smith (kajiya)
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Utkarsh, or anyone else affected,

Accepted openssh into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.4p1-6ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Impish):
status: New → Fix Committed
tags: added: verification-needed verification-needed-impish
Revision history for this message
Robie Basak (racb) wrote :

Hello Utkarsh, or anyone else affected,

Accepted openssh into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.4p1-5ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssh (Ubuntu Hirsute):
status: New → Fix Committed
tags: added: verification-needed-hirsute
Changed in openssh (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Hello Utkarsh, or anyone else affected,

Accepted openssh into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.2p1-4ubuntu0.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssh/1:8.2p1-4ubuntu0.4)

All autopkgtests for the newly accepted openssh (1:8.2p1-4ubuntu0.4) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (ppc64el)
nova/2:21.2.3-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssh/1:8.4p1-5ubuntu1.2)

All autopkgtests for the newly accepted openssh (1:8.4p1-5ubuntu1.2) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/3.0.1-2 (armhf)
diffoscope/171 (armhf)
dropbear/2020.81-3 (armhf)
libssh/0.9.5-1ubuntu0.1 (armhf)
libnetapp-perl/500.002-2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Hello,

All the 2 (for Focal) and 5 (for Hirsute) tests were re-triggered and are passing now, so there's no real regression. We'll proceed with the verification from our end shortly. TIA! \o/

Revision history for this message
Chloé Smith (kajiya) wrote :

Hey everyone,

I can confirm the fix has been tested by our friends at Google (Anthos) for Focal 20.04, using the same patch used in 1:8.2p1-4ubuntu0.4 but *not* by using the package from focal-proposed itself.
Hopefully this still suffices? Please let me know if not and I'll re-run the verification again using an instance pulling from focal-proposed.

[RATIONALE]

Need SSH to authenticate a ``HostCertificate`` and an SSH agent that holds the corresponding host private key.

The sshd_config has the following directives:

-- HostCertificate the public host certificate whose public key matches the private key stored in the ssh agent

-- HostKey the public key of the host keypair

-- HostKeyAgent the socket of the ssh agent that holds the host private key

Before the patch, this combination didn't work - even though it authenticated successfully the setup behaved as if ``HostCertificate`` was never configured (i.e. it authenticated using only the public key and the private key in the ssh agent).

[VERIFICATION OF FIX]
````
sh-agent -a /path/agent-socket
SSH_AUTH_SOCK=/path/agent-socket ssh-add -k /path/hostkey
````

Then ran ``sshd`` with:

````
HostCertificate /path/hostkey-cert.pub
HostKey /path/hostkey.pub
HostKeyAgent /path/agent-socket
````

Then configured the CA trust anchor on the client's side.
(localhost was used, but it would be the same if a second host is used as a client)

````
ssh -vv localhost
````

shows the host certificate was seen and used.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Chris Halse Rogers (raof) wrote :

For SRU verification we require that the actual package from the -proposed archive is tested. This is to avoid the chance that the patch is mis-applied in the package uploaded to the archive, or that the archive build environment differs in a way that affects the patch.

We'll also need this to be verified for Hirsute and Impish before we can release it for Focal, to ensure that upgrades do not regress this bug.

Revision history for this message
Chloé Smith (kajiya) wrote :

Hirsute verification

[INSTALLED PKG VERSION]
kajiya@chloe-HAL:~/Documents/work$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.4p1-5ubuntu1.2
  Candidate: 1:8.4p1-5ubuntu1.2
  Version table:
 *** 1:8.4p1-5ubuntu1.2 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.4p1-5ubuntu1.1 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute-updates/main amd64 Packages
     1:8.4p1-5ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu hirsute/main amd64 Packages

[PROCEDURE]
Create the keys/certs needed
``ssh-keygen -t rsa -b 4096 -f host_ca -C host_ca`` (no passphrase)
``ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa``
``ssh-keygen -s host_ca -I localhost -h -n localhost -V +52w ssh_host_rsa_key.pub``

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running
``ssh -vv localhost 2>&1 | grep "Server host certificate"`` gives

ssh -vv kajiya@localhost 2>&1 | grep "Server host certificate"
debug1: Server host certificate: <email address hidden> SHA256:ufStWAPad1IQ08xMPM1iF4u4JHEaeAuQcD3qoe8yJ9A, serial 0 ID "localhost" CA ssh-rsa SHA256:3iVQ6wcBeoRO3S12jO8K34Do8HbVTPxiBp3rNzCngGc valid from 2022-01-05T17:20:00 to 2023-01-04T17:21:17
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

Revision history for this message
Chloé Smith (kajiya) wrote (last edit ):

Impish verification

[INSTALLED PKG VERSION]
kajiya@chloe-HAL:~$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.4p1-6ubuntu2.1
  Candidate: 1:8.4p1-6ubuntu2.1
  Version table:
 *** 1:8.4p1-6ubuntu2.1 400
        400 http://gb.archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
        400 http://archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.4p1-6ubuntu2 500
        500 http://gb.archive.ubuntu.com/ubuntu impish/main amd64 Packages

[PROCEDURE]
Create the keys/certs needed
``ssh-keygen -t rsa -b 4096 -f host_ca -C host_ca`` (no passphrase)
``ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa``
``ssh-keygen -s host_ca -I localhost -h -n localhost -V +52w ssh_host_rsa_key.pub``

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running
ssh -vv kajiya@localhost 2>&1 | grep "Server host certificate" gives

debug1: Server host certificate: <email address hidden> SHA256:pprTqBvT2oazgTsfPF+RD47ca/W1U4JCgq5fl7m1LkA, serial 0 ID "localhost" CA ssh-rsa SHA256:l3PYuQBJMLruGeASt+BKEDGLDlk5NHx59cwW6/Qgzs4 valid from 2022-01-05T22:11:00 to 2023-01-04T22:12:07
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

Revision history for this message
Chloé Smith (kajiya) wrote :

Focal verification

[INSTALLED PKG VERSION]
chlo@BIG-HAL:~$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.2p1-4ubuntu0.4
  Candidate: 1:8.2p1-4ubuntu0.4
  Version table:
 *** 1:8.2p1-4ubuntu0.4 400
        400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.2p1-4ubuntu0.3 500
        500 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1:8.2p1-4ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     1:8.2p1-4 500
        500 http://gb.archive.ubuntu.com/ubuntu focal/main amd64 Packages

[PROCEDURE]
Create the keys/certs needed
``ssh-keygen -t rsa -b 4096 -f host_ca -C host_ca`` (no passphrase)
``ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa``
``ssh-keygen -s host_ca -I localhost -h -n localhost -V +52w ssh_host_rsa_key.pub``

Copied ssh_host_rsa_key* files over to /etc/ssh and added the following to /etc/ssh/sshd_config
``HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub``

Restarted sshd using systemctl and added ``@cert-authority localhost ssh-rsa abcdefg`` (ssh-rsa abcdefg is the contents of host_ca.pub) to ~/.ssh/known_hosts

Finally, running

ssh -vv chlo@localhost 2>&1 | grep "Server"
debug1: Server host certificate: <email address hidden> SHA256:s2gq1xBSdetCarwElgQd0NbjJbiE3iLDxFtJqDhBFF4, serial 0 ID "localhost" CA ssh-rsa SHA256:v8ZgezKD9Zw/Ns8I0W6mfvxCAo9jv3WznUYAFhfPfCU valid from 2022-01-05T22:46:00 to 2023-01-04T22:47:11
debug2: Server host certificate hostname: localhost

which tells us the certificate was seen and used

tags: added: verification-done-hirsute verification-done-impish
removed: verification-needed-hirsute verification-needed-impish
Revision history for this message
Chloé Smith (kajiya) wrote :

Apologies, I changed the tags *after* posting all the comments just to make it more confusing...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:8.4p1-6ubuntu2.1

---------------
openssh (1:8.4p1-6ubuntu2.1) impish; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

 -- Utkarsh Gupta <email address hidden> Fri, 26 Nov 2021 17:25:36 +0530

Changed in openssh (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:8.4p1-5ubuntu1.2

---------------
openssh (1:8.4p1-5ubuntu1.2) hirsute; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

 -- Chloé S <email address hidden> Wed, 01 Dec 2021 14:12:42 +0000

Changed in openssh (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssh - 1:8.2p1-4ubuntu0.4

---------------
openssh (1:8.2p1-4ubuntu0.4) focal; urgency=medium

  * d/p/match-host-certs-w-public-keys.patch: Add patch
    to match host certificates agianst host public keys.
    (LP: #1952421)

 -- Chloé S <email address hidden> Thu, 02 Dec 2021 22:38:52 +0000

Changed in openssh (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers