openssh 8.9p1 for Jammy auth2-gss patch for gssapi-keyex method is slightly wrong

[ake sandgren]

[ Impact ]

The gssapi-keyex authentication mechanism has been inadvertently broken in openssh. It comes from a distro patch[1], and while the patch still applied, it was no longer correct.

Without the fix, sshd will fail to start if gssapi-keyex is listed in the AuthenticationMethods of the server, and if not, sshd will still start, but gssapi-keyex will not be available.

[ Test Plan ]

This update, besides fixing the patch, also adds a new autopkgtest to the package, which tests both gssapi-with-mic ("normal" gssapi, which is not affected by this bug), and gssapi-keyex, which, before this update, did not work.

The test plan is to run the new ssh-gssapi autopkgtest and verify it succeeds.

[ Where problems could occur ]

ssh is a critical piece of infrastructure, and problems with it could have catastrophic consequences. The service itself has a test command before it starts up to verify the syntax of the config file, but that test is not applied on shutdown, so a restart with an invalid config file could still leave sshd dead.

The patch adds a change to an authentication structure, but that change is already present in the upstream code, and we are just updating it in the new gssapi-keyex code (introduced by the distro[1] patch, already present). Therefore, mistakes here should manifest themselves just in the gssapi-keyex code, which wasn't working anyway. Effectively, though, we are enabling a new authentication mechanism in sshd, one that was not supposed to have been removed, but was broken by mistake.

[ Other Info ]

The fact no-one noticed this problem for more than two years could be telling that there are not many users of this authentication mechanism out there. The same applies to debian: it has also been broken for a while there. Maybe we should drop it for future ubuntu releases, since upstream refuses to take it in.

1. https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/patches/gssapi.patch

[ Original Description ]

The Authmethod struct now have 4 entries but the initialization of the method_gsskeyex in the debian/patches/gssapi.patch only have 3 entries.

The struct was changed in upstream commit dbb339f015c33d63484261d140c84ad875a9e548 as
===
@@ -104,7 +104,8 @@ struct Authctxt {

 struct Authmethod {
        char *name;
- int (*userauth)(struct ssh *);
+ char *synonym;
+ int (*userauth)(struct ssh *, const char *);
        int *enabled;
 };

===

The incorrect code does
===
+Authmethod method_gsskeyex = {
+ "gssapi-keyex",
+ userauth_gsskeyex,
+ &options.gss_authentication
+};
===
but should have a NULL between the "gssapi-keyex" string and userauth_gsskeyex

This is now (change from Focal) causing gssapi-keyex to be disabled.

===
lsb_release -rd
Description: Ubuntu 22.04.3 LTS
Release: 22.04

===
apt-cache policy openssh-server
openssh-server:
  Installed: 1:8.9p1-3ubuntu0.6
  Candidate: 1:8.9p1-3ubuntu0.6
  Version table:
 *** 1:8.9p1-3ubuntu0.6 500
        500 http://faiserver.hpc2n.umu.se/mirrors/ubuntu/ubuntu jammy-updates/main amd64 Packages
        500 http://faiserver.hpc2n.umu.se/mirrors/ubuntu/ubuntu jammy-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1:8.9p1-3 500
        500 http://faiserver.hpc2n.umu.se/mirrors/ubuntu/ubuntu jammy/main amd64 Packages

===

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paride Legovini (paride)]

Hello and thanks for this bug report. The analysis looks sensible to me, but I'm not really familiar with gss. To better understand the situation I have a couple of questions:

- Does this mean that gss is unusable in Jammy at the moment? AFAICT this is the only bug report about it, so I would be surprised.

- Are you able to test openssh from newer Ubuntu releases, ideally including Noble (the next LTS, currently in development)? The gss patch has been updated since Jammy, the last time on:

Last-Updated: 2023-12-18

Thanks!

[ake sandgren (ake-sandgren)]

Verifying this should be fairly simple.

Look at the definition of Authmethod in auth.h and compare to how method_gssapi is initialized compared to method_gsskeyex.

As for it being the only report it is only "AuthenticationMethods gssapi-keyex" that is not working.
We have "AuthenticationMethods gssapi-keyex gssapi-with-mic" so on Jammy it still works but we get complaints in the log, like this:
===
error: Disabled method "gssapi-keyex" in AuthenticationMethods list "gssapi-keyex"
Authentication methods list "gssapi-keyex" contains disabled method, skipping
===

Regarding Noble, the patch for this in openssh_9.6p1-3ubuntu1.debian.tar.xz is still having the same problem with the initialization of method_gsskeyex.

===
@@ -333,6 +377,12 @@ input_gssapi_mic(int type, u_int32_t plen, struct ssh *ssh)
        return 0;
 }

+Authmethod method_gsskeyex = {
+ "gssapi-keyex",
+ userauth_gsskeyex,
+ &options.gss_authentication
+};
+
 Authmethod method_gssapi = {
        "gssapi-with-mic",
        NULL,
===

Note that there is still only three arguments in the init of method_gsskeyex vs the required four.

[Mitchell Dzurick (mitchdz)]

Thanks for the additional info Ake!

Do you happen to have simple steps you could share to help reproduce the issue?

[Robie Basak (racb)]

I'm flagging this as Critical. It sounds like everyone agrees that the distro patch we're carrying is bad. I think it's possible that it's bad in quite a serious way, so we should investigate immediately without delay until we've understood the severity of this, especially because by carrying the distro patch we should be maintaining it. If it turns out to be benign then we can downgrade the bug importance.

[ake sandgren (ake-sandgren)]

We have this in sshd_config
===
Match User root
        GSSAPIAuthentication yes
        PasswordAuthentication no
        KbdInteractiveAuthentication no
        PubkeyAuthentication no
        AuthenticationMethods gssapi-keyex gssapi-with-mic
===

Grab a kerberos root ticket and do ssh as root to a node with that config, it will spew out
===
error: Disabled method "gssapi-keyex" in AuthenticationMethods list "gssapi-keyex"
Authentication methods list "gssapi-keyex" contains disabled method, skipping
===
in the syslog.

It's as simple as I can make it.

But as I said before, this is an obvious bug in the patch since it hasn't been updated to match the change in auth.h

[Andreas Hasenack (ahasenack)]

https://src.fedoraproject.org/rpms/openssh/c/c04e468b07b38471377fc7a648e1737021ea7148

[Andreas Hasenack (ahasenack)]

Prepping builds, and I also want to add an autopkgtest for this.

[Andreas Hasenack (ahasenack)]

Quick test with https://launchpad.net/~ahasenack/+archive/ubuntu/openssh-gsskeyex-2053146/+packages on jammy (but there are builds for other releases too), seems to work:

Mar 13 20:52:58 j-keyex sshd[1638]: Authorized to ubuntu, krb5 principal andreas@LOWTECH (krb5_kuserok)
Mar 13 20:52:58 j-keyex sshd[1638]: Accepted gssapi-keyex for ubuntu from 10.0.102.1 port 48450 ssh2: andreas@LOWTECH

[Andreas Hasenack (ahasenack)]

I have an autopkgtest for gssapi, adding one now for keyex.

[Colin Watson (cjwatson)]

I fixed this in Debian today in https://salsa.debian.org/ssh-team/openssh/-/commit/0947dd466d64cabfb527d8326e2507f473373a32, uploaded as part of 1:9.7p1-1. You could possibly just merge 1:9.7p1-1 into noble since it's mostly a bug-fix release, but failing that you could cherry-pick the relevant change easily enough.

[Andreas Hasenack (ahasenack)]

I think you missed the extra arg to userauth_gsskeyex()

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:9.6p1-3ubuntu11

---------------
openssh (1:9.6p1-3ubuntu11) noble; urgency=medium

  * d/t/ssh-gssapi: make the test a bit more rebust (LP: #2058276):
    - deal with return codes
    - match a more specific success expression from the logs
    - add klist output in the case of failure

 -- Andreas Hasenack <email address hidden> Mon, 18 Mar 2024 10:25:15 -0300

[Timo Aaltonen (tjaalton)]

Hello ake, or anyone else affected,

Accepted openssh into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.3p1-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello ake, or anyone else affected,

Accepted openssh into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.9p1-3ubuntu0.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssh (1:9.3p1-1ubuntu3.3) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.52.0-1 (arm64, i386)
libssh/0.10.5-3ubuntu1.2 (i386)
piuparts/1.1.7 (amd64)
pkg-perl-tools/0.76 (arm64)
vagrant/2.3.4+dfsg-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssh (1:8.9p1-3ubuntu0.7) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

backdoor-factory/unknown (armhf)
gvfs/1.48.2-0ubuntu1 (arm64)
libssh/0.9.6-2ubuntu0.22.04.3 (i386)
piuparts/1.1.5 (amd64)
sbuild/0.81.2ubuntu6 (arm64, ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[ake sandgren (ake-sandgren)]

openssh-server_8.9p1-3ubuntu0.7_amd64.deb does fix the gssapi-keyex problem for us on jammy

Syslog output is as expected

===
2024-04-08T08:09:53.608275+02:00 somehost sshd[169530]: Authorized to root, krb5 principal <email address hidden> (krb5_kuserok)
2024-04-08T08:09:53.619114+02:00 somehost sshd[169530]: Accepted gssapi-keyex for root from 1.2.3.4 port 60232 ssh2: <email address hidden>

===

[Andreas Hasenack (ahasenack)]

Jammy verification

In all architectures (except i386, which is a known failure everywhere) the new ssh-gssapi test passed.

Here is the run on amd64[1]:
3438s autopkgtest [16:33:21]: test ssh-gssapi: [-----------------------
3438s ## Setting up test environment
3438s ## Creating Kerberos realm EXAMPLE.FAKE
3438s Loading random data
3438s Initializing database '/var/lib/krb5kdc/principal' for realm 'EXAMPLE.FAKE',
3438s master key name '<email address hidden>'
3438s ## Creating principals
3438s Authenticating as principal <email address hidden> with password.
3438s Principal "<email address hidden>" created.
3438s Authenticating as principal <email address hidden> with password.
3438s Principal "<email address hidden>" created.
3438s ## Extracting service principal host/sshd-gssapi.example.fake
3438s Authenticating as principal <email address hidden> with password.
3438s Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
3438s Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
3438s ## Adjusting /etc/krb5.conf
3438s ## TESTS
3438s
3438s ## TEST test_gssapi_login
3438s ## Configuring sshd for gssapi-with-mic authentication
3438s ## Restarting ssh
3438s ## Obtaining TGT
3438s Password for <email address hidden>:
3438s Ticket cache: FILE:/tmp/krb5cc_0
3438s Default principal: <email address hidden>
3438s
3438s Valid starting Expires Service principal
3438s 04/05/24 16:33:20 04/06/24 02:33:20 <email address hidden>
3438s renew until 04/06/24 16:33:20
3438s
3438s ## ssh'ing into localhost using gssapi-with-mic auth
3438s Warning: Permanently added 'sshd-gssapi.example.fake' (ED25519) to the list of known hosts.
3439s Fri Apr 5 16:33:21 UTC 2024
3439s
3439s ## checking that we got a service ticket for ssh (host/)
3439s 04/05/24 16:33:21 04/06/24 02:33:20 host/sshd-gssapi.example.fake@
3439s Ticket server: <email address hidden>
3439s
3439s ## Checking ssh logs to confirm gssapi-with-mic auth was used
3439s Apr 05 16:33:21 sshd-gssapi.example.fake sshd[1518]: Accepted gssapi-with-mic for testuser1457 from 127.0.0.1 port 50668 ssh2: <email address hidden>
3439s ## PASS test_gssapi_login
3439s
3439s ## TEST test_gssapi_keyex_login
3439s ## Configuring sshd for gssapi-keyex authentication
3439s ## Restarting ssh
3439s ## Obtaining TGT
3439s Password for <email address hidden>:
3439s Ticket cache: FILE:/tmp/krb5cc_0
3439s Default principal: <email address hidden>
3439s
3439s Valid starting Expires Service principal
3439s 04/05/24 16:33:21 04/06/24 02:33:21 <email address hidden>
3439s renew until 04/06/24 16:33:21
3439s
3439s ## ssh'ing into localhost using gssapi-keyex auth
3439s Fri Apr 5 16:33:21 UTC 2024
3439s
3439s ## checking that we got a service ticket for ssh (host/)
3439s 04/05/24 16:33:21 04/06/24 02:33:21 host/sshd-gssapi.example.fake@
3439s Ticket server: <email address hidden>
3439s
3439s ## Checking ssh logs...

[Andreas Hasenack (ahasenack)]

Mantic verification

In all architectures, except i386, the new test passed.

Here is a log from the amd64 run[1]:

4333s autopkgtest [16:47:27]: test ssh-gssapi: [-----------------------
4333s ## Setting up test environment
4333s ## Creating Kerberos realm EXAMPLE.FAKE
4333s Initializing database '/var/lib/krb5kdc/principal' for realm 'EXAMPLE.FAKE',
4333s master key name '<email address hidden>'
4333s ## Creating principals
4333s Authenticating as principal <email address hidden> with password.
4333s Principal "<email address hidden>" created.
4333s Authenticating as principal <email address hidden> with password.
4333s Principal "<email address hidden>" created.
4333s ## Extracting service principal host/sshd-gssapi.example.fake
4333s Authenticating as principal <email address hidden> with password.
4333s Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
4333s Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
4333s ## Adjusting /etc/krb5.conf
4333s ## TESTS
4333s
4333s ## TEST test_gssapi_login
4333s ## Configuring sshd for gssapi-with-mic authentication
4333s ## Restarting ssh
4333s ## Obtaining TGT
4333s Password for <email address hidden>:
4333s Ticket cache: FILE:/tmp/krb5cc_0
4333s Default principal: <email address hidden>
4333s
4333s Valid starting Expires Service principal
4333s 04/05/24 16:47:27 04/06/24 02:47:27 <email address hidden>
4333s renew until 04/06/24 16:47:27
4333s
4333s ## ssh'ing into localhost using gssapi-with-mic auth
4333s Warning: Permanently added 'sshd-gssapi.example.fake' (ED25519) to the list of known hosts.
4334s Fri Apr 5 16:47:27 UTC 2024
4334s
4334s ## checking that we got a service ticket for ssh (host/)
4334s 04/05/24 16:47:27 04/06/24 02:47:27 host/sshd-gssapi.example.fake@
4334s Ticket server: <email address hidden>
4334s
4334s ## Checking ssh logs to confirm gssapi-with-mic auth was used
4334s Apr 05 16:47:27 sshd-gssapi.example.fake sshd[1688]: Accepted gssapi-with-mic for testuser1620 from 127.0.0.1 port 44922 ssh2: <email address hidden>
4334s ## PASS test_gssapi_login
4334s
4334s ## TEST test_gssapi_keyex_login
4334s ## Configuring sshd for gssapi-keyex authentication
4334s ## Restarting ssh
4334s ## Obtaining TGT
4334s Password for <email address hidden>:
4334s Ticket cache: FILE:/tmp/krb5cc_0
4334s Default principal: <email address hidden>
4334s
4334s Valid starting Expires Service principal
4334s 04/05/24 16:47:28 04/06/24 02:47:28 <email address hidden>
4334s renew until 04/06/24 16:47:28
4334s
4334s ## ssh'ing into localhost using gssapi-keyex auth
4334s Fri Apr 5 16:47:28 UTC 2024
4334s
4334s ## checking that we got a service ticket for ssh (host/)
4334s 04/05/24 16:47:28 04/06/24 02:47:28 host/sshd-gssapi.example.fake@
4334s Ticket server: <email address hidden>
4334s
4334s ## Checking ssh logs to confirm gssapi-keyex auth was used
4334s Apr 05 16:47:28 ssh...

[Robie Basak (racb)]

It's not clear to me if a simple "ssh -Snone localhost" is covered by the autopkgtests, so I did that manually, testing without -proposed first, and ensuring to run "sudo systemctl restart ssh" after upgrading to -proposed to ensure that I'm definitely hitting the daemon from -proposed.

Success on: 1:8.9p1-3ubuntu0.7 on Jammy and 1:9.3p1-1ubuntu3.3 on Mantic.

My commands were:

lxc launch ubuntu:jammy foo
lxc exec foo bash
login -f ubuntu
ssh-keygen # and set no passphrase
cd .ssh
cat id_rsa.pub >> authorized_keys
ssh -Snone localhost
exit
sudo add-apt-repository -p proposed
apt install -t jammy-proposed openssh-server
sudo systemctl restart ssh
ssh -Snone localhost
exit
apt policy openssh-server

(and the equivalent for Mantic)

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:9.3p1-1ubuntu3.3

---------------
openssh (1:9.3p1-1ubuntu3.3) mantic; urgency=medium

  * d/p/gssapi.patch: fix method_gsskeyex structure and
    userauth_gsskeyex function regarding changes introduced in upstream
    commit dbb339f015c33d63484261d140c84ad875a9e548 ("prepare for
    multiple names for authmethods") (LP: #2053146)
  * d/t/{ssh-gssapi,util}: ssh-gssapi DEP8 test for gssapi-with-mic
    and gssapi-keyex authentication methods

 -- Andreas Hasenack <email address hidden> Fri, 15 Mar 2024 17:25:30 -0300

[Robie Basak (racb)]

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:8.9p1-3ubuntu0.7

---------------
openssh (1:8.9p1-3ubuntu0.7) jammy; urgency=medium

  * d/p/gssapi.patch: fix method_gsskeyex structure and
    userauth_gsskeyex function regarding changes introduced in upstream
    commit dbb339f015c33d63484261d140c84ad875a9e548 ("prepare for
    multiple names for authmethods") (LP: #2053146)
  * d/t/{ssh-gssapi,util}: ssh-gssapi DEP8 test for gssapi-with-mic
    and gssapi-keyex authentication methods

 -- Andreas Hasenack <email address hidden> Fri, 15 Mar 2024 17:28:22 -0300