[SRU] SSH pubkey authetication fails when GSSAPI enabled

[Moritz Carmesin]

[ Impact ]

 * Login with publickey fails when openssh server is configured to use GSSAPI authentication, too. Error: "sign_and_send_pubkey: internal error: initial hostkey not recorded"

 * To trigger it, one needs to (a) perform a successful GSSAPI key exchange, (b) attempt public key authentication.

 * In addition, the client and the server must both have the hostbound authentication protocol extension enabled for the problem to manifest itself (On by default).

 * This is not a very common combination, but it can happen if one has Kerberos credentials for the correct realm but the wrong user, and a private key for the right user.

 * This SRU fixes this by adding an additional "ssh->kex->initial_hostkey != NULL" check in sshconnect2.c:sign_and_send_pubkey(), as suggested by upstream in https://bugzilla.mindrot.org/show_bug.cgi?id=3406 (comment 2).

[ Test Plan ]

 The reproducer was codified in autopkgtests, thanks to Colin Watson!

 * Make sure to have the latest debian/tests/ssh-gssapi test case (included as of 1:9.9p1-2, and shipped as part of this SRU), especially the delta described in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2028282/+attachment/5845545/+files/dep8-verifier.diff

 * Execute the "ssh-gssapi" dep8 test:
$ autopkgtest -U openssh --apt-pocket=proposed=src:openssh --test-name=ssh-gssapi -- lxd autopkgtest/ubuntu/oracular/amd64

 * Confirm the log contains 3 login attempts, with the final one using the "publickey" authentication method ("Accepted publickey for testuser..."):
"""
## Checking ssh logs to confirm publickey auth was used
Dec 14 22:44:16 sshd-gssapi.example.fake sshd-session[2213]: Accepted publickey for testuser2020-2 from 127.0.0.1 port 43364 ssh2: ED25519 SHA256:7vF3468XCZOawompwDThLsGsnPoUaP5Ki/3KaQLq/2M
## PASS test_gssapi_keyex_pubkey_fallback
"""

[ Test Plan 2 ]

 * In addition to the codified test for this specific issue, we want to confirm normal password and publickey login are still working as expected.

 * Enable "PasswordAuthentication yes" in /etc/ssh/sshd_config & restart ssh.service

 * Login using password, confirm success

 * Copy public key over to system-under-test

 * Enable "PubkeyAuthentication yes" in /etc/ssh/sshd_config & restart

 * Login using private key, confirm success

[ Where problems could occur ]

 * This SRU tweaks the authentication logic of OpenSSH, therefore it's a high-impact change. If something goes wrong, it could lock people out of their remote machines.

 * The change has been deployed to Debian testing and Ubuntu Plucky since October 2024, without major issues raised.

 * I've added "[ Test Plan 2 ]" to confirm normal publickey & password login is still working as expected

[ Other Info ]

 * Fixed as of 1:9.9p1-2 (e.g. in Plucky)

 * Rejected upstream, due to being a bug in the Debian delta:
   https://bugzilla.mindrot.org/show_bug.cgi?id=3406

 * Fixed in Debian by Colin Watson:
   https://salsa.debian.org/ssh-team/openssh/-/commit/7d291bb

=== original bug report ===
Since the upgrade from Ubuntu 20.04 to 22.04 the SSH login via a SSH pubkey to our servers fails, while password and kerberos are still working.

$ssh user@server
sign_and_send_pubkey: internal error: initial hostkey not recorded

This seem related to the bugreport at openssh: https://bugzilla.mindrot.org/show_bug.cgi?id=3406

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: openssh-server 1:8.9p1-3ubuntu0.1
ProcVersionSignature: Ubuntu 5.15.0-76.83-generic 5.15.99
Uname: Linux 5.15.0-76-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
Date: Thu Jul 20 17:25:01 2023
InstallationDate: Installed on 2020-08-24 (1060 days ago)
InstallationMedia: Ubuntu-Server 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
SourcePackage: openssh
UpgradeStatus: Upgraded to jammy on 2023-07-20 (0 days ago)

[Steve Langasek (vorlon)]

I can confirm this behavior here.

I've found I can work around it by unsetting KRB5CCNAME in the client environment.

[Moritz Carmesin (carmesinus)]

Unsetting KRB5CCNAME breaks the Kerbreros based login, so it is really just an emergency workaround.

[Steve Langasek (vorlon)]

On Wed, Jul 26, 2023 at 07:24:37AM -0000, Moritz Carmesin wrote:
> Unsetting KRB5CCNAME breaks the Kerbreros based login, so it is really
> just an emergency workaround.

You can unset it for just the ssh process that needs non-Kerberos login.

[Markus Rexhepi-Lindberg (k-markus)]

It appears to be solved in 24.04.

[Colin Watson (cjwatson)]

I think comment #5 must have been a testing error. I can still reproduce this with the latest version in Debian, and I don't see anything relevant when looking through the current Ubuntu patch set.

[Andreas Hasenack (ahasenack)]

From https://bugzilla.mindrot.org/show_bug.cgi?id=3406#c8

Colin committed a fix[1] to debian salsa, and we should grab that as well, and SRU this at first glance.

1. https://salsa.debian.org/ssh-team/openssh/-/commit/7d291bb6319611a01dfa0f56fd161db11547320f

[Lukas Märdian (slyon)]

A reproducer for this got codified as of 1:9.9p1-2 (thanks to Colin Watson). It's therefore included in Plucky and can be easily reproduced on Oracular.

To reproduce, we can use the attached "dep8-verifier.diff" to add corresponding autopkgtest improvements to Oracular, which is then showing the very same "sign_and_send_pubkey: internal error: initial hostkey not recorded" error on login:

$ autopkgtest -U -B . --test-name=ssh-gssapi -- lxd autopkgtest/ubuntu/oracular/amd64
autopkgtest [11:37:29]: starting date and time: 2024-12-16 11:37:29+0100
autopkgtest [11:37:29]: version 5.38ubuntu1~24.04.1
autopkgtest [11:37:29]: host abaconcy; command line: /usr/bin/autopkgtest -U -B . --test-name=ssh-gssapi -- lxd autopkgtest/ubuntu/oracular/amd64
autopkgtest [11:37:39]: testbed dpkg architecture: amd64
autopkgtest [11:37:39]: testbed apt version: 2.9.8
autopkgtest [11:37:39]: @@@@@@@@@@@@@@@@@@@@ test bed setup
[...]
autopkgtest [11:39:24]: test ssh-gssapi: [-----------------------
Generating public/private ed25519 key pair.
Your identification has been saved in /root/.ssh/id_ed25519
Your public key has been saved in /root/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:nfzzPGYVv5pAD0m7vBb9qFZvuo8U1tumvTK4wGEvIsk root@autopkgtest-lxd-coegkp
The key's randomart image is:
+--[ED25519 256]--+
| |
| |
| . |
| o..o .. |
| So+=.o .o|
| . . o =o=o. =|
| E . + =*o+o+|
| . . o++B*O |
| o+oO%+o|
+----[SHA256]-----+
## Setting up test environment
## Creating Kerberos realm EXAMPLE.FAKE
Initializing database '/var/lib/krb5kdc/principal' for realm 'EXAMPLE.FAKE',
master key name '<email address hidden>'
## Creating principals
Authenticating as principal <email address hidden> with password.
Principal "<email address hidden>" created.
Authenticating as principal <email address hidden> with password.
Principal "<email address hidden>" created.
## Extracting service principal host/sshd-gssapi.example.fake
Authenticating as principal <email address hidden> with password.
Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
Entry for principal host/sshd-gssapi.example.fake with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
## Adjusting /etc/krb5.conf
## TESTS

## TEST test_gssapi_login
## Configuring sshd for gssapi-with-mic authentication
## Restarting ssh
## Obtaining TGT
Password for <email address hidden>:
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: <email address hidden>

Valid starting Expires Service principal
12/16/24 10:39:25 12/16/24 20:39:25 <email address hidden>
 renew until 12/17/24 10:39:25

## ssh'ing into localhost using gssapi-with-mic auth
Warning: Permanently added 'sshd-gssapi.example.fake' (ED25519) to the list of known hosts.
Mon Dec 16 10:39:25 UTC 2024

## checking that we got a service ticket for ssh (host/)
12/16/24 10:39:25 12/16/24 20:39:25 host/sshd-gssapi.example.fake@
 Ticket server: <email address hidden>

## Checking ssh logs to confi...

[Lukas Märdian (slyon)]

The Plucky autopkgtest logs from https://autopkgtest.ubuntu.com/packages/openssh already show:

"""
## Checking ssh logs to confirm publickey auth was used
Dec 14 22:44:16 sshd-gssapi.example.fake sshd-session[2213]: Accepted publickey for testuser2020-2 from 127.0.0.1 port 43364 ssh2: ED25519 SHA256:7vF3468XCZOawompwDThLsGsnPoUaP5Ki/3KaQLq/2M
## PASS test_gssapi_keyex_pubkey_fallback
"""

[Lukas Märdian (slyon)]

Uploaded into Oracular for SRU review.

Holding back the upload to Noble (and Jammy), for now, due to a pending SRU regression in 1:9.6p1-3ubuntu13.7. I don't want this to get entangled. Maybe we should bundle it on top, though?

[Lukas Märdian (slyon)]

enr0n says:
"Please leave noble alone for now. @ahasenack and I have been monitoring the regression report and hoping for more feedback, but we still need to decide how to move forward. In any case, that won't be solved before EOY."

[Timo Aaltonen (tjaalton)]

Hello Moritz, or anyone else affected,

Accepted openssh into oracular-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.7p1-7ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-oracular to verification-done-oracular. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-oracular. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssh (1:9.7p1-7ubuntu4.1) for oracular have finished running.
The following regressions have been reported in tests triggered by the package:

diffoscope/unknown (ppc64el)
fence-agents/4.15.0-1ubuntu2 (ppc64el)
glome/unknown (ppc64el)
goss/0.4.8-1 (amd64)
libnet-scp-perl/0.08-4 (amd64)
openssh/1:9.7p1-7ubuntu4.1 (amd64)
piuparts/1.4.3 (amd64, i386)
systemd/256.5-2ubuntu3.1 (arm64, ppc64el)
vorta/0.9.1-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/oracular/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lukas Märdian (slyon)]

Autopkgtests are looking good for openssh 1:9.7p1-7ubuntu4.1 from oracular-proposed. The intermittent failures from comment #13 have all been resolved.

https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/amd64/o/openssh/20241218_102717_96d50@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/arm64/o/openssh/20241217_173242_f3550@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/armhf/o/openssh/20241217_172946_d43fe@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/ppc64el/o/openssh/20241217_201054_4289f@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/s390x/o/openssh/20241217_203414_b4f03@/log.gz

All of them show 3 login attempts with the final one using privatekey authentication:

"""
1140s ## Checking ssh logs to confirm gssapi-with-mic auth was used
1140s Dec 18 10:25:16 sshd-gssapi.example.fake sshd[1626]: Accepted gssapi-with-mic for testuser1525 from 127.0.0.1 port 34338 ssh2: <email address hidden>
1140s ## PASS test_gssapi_login
[...]
1140s ## Checking ssh logs to confirm gssapi-keyex auth was used
1140s Dec 18 10:25:16 sshd-gssapi.example.fake sshd[1679]: Accepted gssapi-keyex for testuser1525 from 127.0.0.1 port 34352 ssh2: <email address hidden>
1140s ## PASS test_gssapi_keyex_login
[...]
1140s ## Checking ssh logs to confirm publickey auth was used
1140s Dec 18 10:25:17 sshd-gssapi.example.fake sshd[1718]: Accepted publickey for testuser1525-2 from 127.0.0.1 port 34358 ssh2: ED25519 SHA256:LzS5GWVNXlP4zbwYWz2Dfd3CzLN7Z2S94q2Jpm66l7U
1140s ## PASS test_gssapi_keyex_pubkey_fallback
"""

[Lukas Märdian (slyon)]

I tested openssh (1:9.7p1-7ubuntu4.1) from oracular proposed, according to TestCase1 (comment #14) and TestCase2 (this comment). All looking good!

[ Test Case 2 ]

###Set up an Oracular LXD container & install openssh-server from proposed:
root@oosru:~# apt list *openssh-server*
openssh-server/oracular-proposed,now 1:9.7p1-7ubuntu4.1 amd64 [installed]
root@oosru:~# adduser test
info: Adding user `test' ...
info: Selecting UID/GID from range 1000 to 59999 ...
info: Adding new group `test' (1001) ...
info: Adding new user `test' (1001) with group `test (1001)' ...
info: Creating home directory `/home/test' ...
info: Copying files from `/etc/skel' ...
New password: [test]
Retype new password: [test]
passwd: password updated successfully

### Adopt SSH config & restart service
root@oosru:~# vim /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
root@oosru:~# grep -R PasswordAuthentication /etc/ssh/
/etc/ssh/ssh_config:# PasswordAuthentication yes
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication yes
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
root@oosru:~# systemctl restart ssh.service
root@oosru:~# ip a show eth0
548: eth0@if549: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:d1:bc:18 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.238.94.46/24 metric 100 brd 10.238.94.255 scope global dynamic eth0
       valid_lft 3221sec preferred_lft 3221sec
    inet6 fd42:7213:f20e:bd74:216:3eff:fed1:bc18/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fed1:bc18/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

### From the host (password login OK):
$ ssh test@10.238.94.46
test@10.238.94.46's password: [test]

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

test@oosru:~$ ssh-import-id-lp slyon
2025-01-07 09:38:45,613 INFO Authorized key ['4096', 'SHA256:sciOAYEEOgZuev6e/fxLpojXxsiZsJPzn1Jk8LaYvVg', '<email address hidden>', '(RSA)']
2025-01-07 09:38:45,613 INFO [1] SSH keys [Authorized]

### From the host (pubkey login OK):
$ ssh -i ~/.ssh/canonical_id_rsa test@10.238.94.46
Enter passphrase for key '/home/lukas/.ssh/canonical_id_rsa':
Welcome to Ubuntu 24.10 (GNU/Linux 6.8.0-49-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

 System information as of Tue Jan 7 09:39:47 UTC 2025

  System load: 0.94
  Usage of /: 89.2% of 195.80GB
  Memory usage: 0%
  Swap usage: 0%
  Temperature: 47.0 C
  Processes: 26
  Users logged in: 0
  IPv4 address for eth0: 10.238.94.46
  IPv6 address for eth0: fd42:7213:f20e:bd...

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:9.7p1-7ubuntu4.1

---------------
openssh (1:9.7p1-7ubuntu4.1) oracular; urgency=medium

  * d/p/gssapi.patch: Fix interaction between gssapi-keyex and pubkey auth
    (LP: #2028282)
    Don't prefer host-bound public key signatures if there was no initial
    host key, as is the case when using GSS-API key exchange.
    Thanks to Colin Watson for providing patches via Debian Salsa (7d291bb)
    + d/t/ssh-gssapi: Fix typo in autopkgtest
    + d/t/ssh-gssapi: Test interaction between gssapi-keyex and pubkey auth.

 -- Lukas Märdian <email address hidden> Mon, 16 Dec 2024 12:49:45 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for openssh has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Timo Aaltonen (tjaalton)]

Hello Moritz, or anyone else affected,

Accepted openssh into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:9.6p1-3ubuntu13.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssh (1:9.6p1-3ubuntu13.9) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

ceph/19.2.0-0ubuntu0.24.04.2 (amd64, s390x)
crmsh/4.6.0-1ubuntu2 (amd64, arm64, armhf, i386, ppc64el, s390x)
libnet-ssh-perl/0.09-5 (s390x)
libnet-ssh-perl/unknown (armhf)
mrbayes/3.2.7a-7build2 (s390x)
nova/3:29.0.1-0ubuntu1.4 (amd64)
openssh/1:9.6p1-3ubuntu13.9 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lukas Märdian (slyon)]

I tested openssh (1:9.6p1-3ubuntu13.9) from noble-proposed, according to "Test Plan 2". Looking good!

[ Test Plan 2 ]

###Set up a Noble LXD container & install openssh-server from proposed:
root@nnsru:~# apt list *openssh-server*
Listing... Done
openssh-server/noble-proposed,now 1:9.6p1-3ubuntu13.9 amd64 [installed]
root@nnsru:~# adduser test
info: Adding user `test' ...
info: Selecting UID/GID from range 1000 to 59999 ...
info: Adding new group `test' (1001) ...
info: Adding new user `test' (1001) with group `test (1001)' ...
info: Creating home directory `/home/test' ...
info: Copying files from `/etc/skel' ...
New password: [test]
Retype new password: [test]
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
 Full Name []:
 Room Number []:
 Work Phone []:
 Home Phone []:
 Other []:
Is the information correct? [Y/n]
info: Adding new user `test' to supplemental / extra groups `users' ...
info: Adding user `test' to group `users' ...

### Adopt SSH config & restart service
root@nnsru:~# vim /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
root@nnsru:~# grep -R PasswordAuthentication /etc/ssh/
/etc/ssh/ssh_config:# PasswordAuthentication yes
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication yes
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
root@nnsru:~# systemctl restart ssh.service
root@nnsru:~# ip a show eth0
199: eth0@if200: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:04:41:7d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.238.94.186/24 metric 100 brd 10.238.94.255 scope global dynamic eth0
       valid_lft 3319sec preferred_lft 3319sec
    inet6 fd42:7213:f20e:bd74:216:3eff:fe04:417d/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe04:417d/64 scope link
       valid_lft forever preferred_lft forever

### From the host (password login OK):
$ ssh test@10.238.94.186
The authenticity of host '10.238.94.186 (10.238.94.186)' can't be established.
ED25519 key fingerprint is SHA256:t2qgSMbZOw1Pm6PXVOL5lXYCtD/JDBUoJZjwygRshNs.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.238.94.186' (ED25519) to the list of known hosts.
test@10.238.94.186's password: [test]

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

test@nnsru:~$ ssh-import-id-lp slyon
2025-03-10 14:17:08,923 INFO Authorized key ['4096', 'SHA256:sciOAYEEOgZuev6e/fxLpojXxsiZsJPzn1Jk8LaYvVg', '<email address hidden>', '(RSA)']
2025-03-10 14:17:08,923 INFO [1] SSH keys [Authorized]

### From the host (pubkey login OK):
$ ssh -i ~/.ssh/canonical_id_rsa tes...

[Lukas Märdian (slyon)]

Autopkgtests are looking good for openssh 1:9.6p1-3ubuntu13.9 from noble-proposed. The intermittent failures from comment #19 have all been resolved, with openssh/s390x and crmsh/any regressed in jammy-updates (unrelated to this update).

Together with the previous comment (#20), this finalizes SRU verification for this bug report.

PASS https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/o/openssh/20250307_164612_f5f9a@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/arm64/o/openssh/20250307_185308_196f2@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/armhf/o/openssh/20250307_182855_7ac4e@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/ppc64el/o/openssh/20250310_143140_2979a@/log.gz
FAIL https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/s390x/o/openssh/20250310_143626_6d8a7@/log.gz

=> For the final openssh/s390x failure: This is regressed in jammy-updates, unrelated to this upload.
The summary shows that the new "ssh-gssapi" test still passes on s390x, which is relevant for SRU verification of this bug:
"""
1735s autopkgtest [14:36:19]: @@@@@@@@@@@@@@@@@@@@ summary
1735s regress FAIL non-zero exit status 2
1735s systemd-socket-activation PASS
1735s sshd-socket-generator PASS
1735s ssh-gssapi PASS
"""

All of them show 3 login attempts with the final one using privatekey authentication:

"""
1733s ## Checking ssh logs to confirm gssapi-with-mic auth was used
1733s Mar 10 14:36:17 sshd-gssapi.example.fake sshd[1678]: Accepted gssapi-with-mic for testuser1562 from 127.0.0.1 port 34648 ssh2: <email address hidden>
1733s ## PASS test_gssapi_login
[...]
1733s ## Checking ssh logs to confirm gssapi-keyex auth was used
1733s Mar 10 14:36:17 sshd-gssapi.example.fake sshd[1730]: Accepted gssapi-keyex for testuser1562 from 127.0.0.1 port 34654 ssh2: <email address hidden>
1733s ## PASS test_gssapi_keyex_login
[...]
1734s ## Checking ssh logs to confirm publickey auth was used
1734s Mar 10 14:36:18 sshd-gssapi.example.fake sshd[1768]: Accepted publickey for testuser1562-2 from 127.0.0.1 port 34658 ssh2: ED25519 SHA256:Y41A5NlL26LzekRtf+/QXlr/SJEnydvTS9aw9yEVT8c
1734s ## PASS test_gssapi_keyex_pubkey_fallback
"""

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:9.6p1-3ubuntu13.9

---------------
openssh (1:9.6p1-3ubuntu13.9) noble; urgency=medium

  [ Lukas Märdian ]
  * d/p/gssapi.patch: Fix interaction between gssapi-keyex and pubkey auth
    (LP: #2028282)
    Don't prefer host-bound public key signatures if there was no initial
    host key, as is the case when using GSS-API key exchange.
    Thanks to Colin Watson for providing patches via Debian Salsa (7d291bb)
    + d/t/ssh-gssapi: Fix typo in autopkgtest
    + d/t/ssh-gssapi: Test interaction between gssapi-keyex and pubkey auth.

  [ Nick Rosbrook ]
  * sshd-socket-generator: do not parse server match config
    (LP: #2076023)

 -- Nick Rosbrook <email address hidden> Mon, 24 Feb 2025 16:25:09 -0500

[Andreas Hasenack (ahasenack)]

Hello Moritz, or anyone else affected,

Accepted openssh into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.9p1-3ubuntu0.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted openssh (1:8.9p1-3ubuntu0.12) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

piuparts/1.1.5 (amd64, arm64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#openssh

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lukas Märdian (slyon)]

re comment #24 piuparts is regressed in -release (since Jammy+): https://autopkgtest.ubuntu.com/packages/piuparts – I triggered migration-reference/0 tests on Jammy to confirm this.

[Lukas Märdian (slyon)]

I tested openssh (1:8.9p1-3ubuntu0.12) from jammy-proposed, according to "Test Plan 2". Looking good!

[ Test Plan 2 ]

###Set up a Jammy LXD container & install openssh-server from proposed:
root@jjsru:~# apt list *openssh-server*
Listing... Done
openssh-server/jammy-proposed,now 1:8.9p1-3ubuntu0.12 amd64 [installed]
root@jjsru:~# adduser test
Adding user `test' ...
Adding new group `test' (1001) ...
Adding new user `test' (1001) with group `test' ...
Creating home directory `/home/test' ...
Copying files from `/etc/skel' ...
New password: [test]
Retype new password: [test]
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
 Full Name []:
 Room Number []:
 Work Phone []:
 Home Phone []:
 Other []:
Is the information correct? [Y/n]

### Adopt SSH config & restart service
root@jjsru:~# vim /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
root@jjsru:~# grep -R PasswordAuthentication /etc/ssh/
/etc/ssh/ssh_config:# PasswordAuthentication yes
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication yes
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
root@jjsru:~# systemctl restart ssh.service
root@jjsru:~# ip a show eth0
41: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:49:68:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.238.94.15/24 metric 100 brd 10.238.94.255 scope global dynamic eth0
       valid_lft 3390sec preferred_lft 3390sec
    inet6 fd42:7213:f20e:bd74:216:3eff:fe49:68fb/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe49:68fb/64 scope link
       valid_lft forever preferred_lft forever

### From the host (password login OK):
$ ssh test@10.238.94.15
The authenticity of host '10.238.94.15 (10.238.94.15)' can't be established.
ED25519 key fingerprint is SHA256:nC8MUedwKPMY/uH6RjxGExIHo06T1w+9o7yblelI/XQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.238.94.15' (ED25519) to the list of known hosts.
test@10.238.94.15's password: [test]

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

test@jjsru:~$ ssh-import-id-lp slyon
2025-04-01 13:24:57,218 INFO Authorized key ['4096', 'SHA256:sciOAYEEOgZuev6e/fxLpojXxsiZsJPzn1Jk8LaYvVg', '<email address hidden>', '(RSA)']
2025-04-01 13:24:57,219 INFO [1] SSH keys [Authorized]

### From the host (pubkey login OK):
$ ssh -i ~/.ssh/canonical_id_rsa test@10.238.94.15
Enter passphrase for key '/home/lukas/.ssh/canonical_id_rsa':
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.8.0-55-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://la...

[Lukas Märdian (slyon)]

Autopkgtests are looking good for openssh 1:8.9p1-3ubuntu0.12 from jammy-proposed.

Together with the previous comment (#26), this finalizes SRU verification for this bug report.

PASS https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/o/openssh/20250327_230556_f645e@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/o/openssh/20250327_231716_503da@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/armhf/o/openssh/20250327_231753_446ba@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/ppc64el/o/openssh/20250401_045019_526cc@/log.gz
PASS https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/s390x/o/openssh/20250327_232902_01be4@/log.gz

All of them show 3 login attempts with the final one using privatekey authentication:

"""
1624s ## Checking ssh logs to confirm gssapi-with-mic auth was used
1624s Mar 27 23:05:38 sshd-gssapi.example.fake sshd[1457]: Accepted gssapi-with-mic for testuser1357 from 127.0.0.1 port 59278 ssh2: <email address hidden>
1624s ## PASS test_gssapi_login
[...]
1625s ## Checking ssh logs to confirm gssapi-keyex auth was used
1625s Mar 27 23:05:38 sshd-gssapi.example.fake sshd[1500]: Accepted gssapi-keyex for testuser1357 from 127.0.0.1 port 59286 ssh2: <email address hidden>
1625s ## PASS test_gssapi_keyex_login
[...]
1625s ## Checking ssh logs to confirm publickey auth was used
1625s Mar 27 23:05:39 sshd-gssapi.example.fake sshd[1535]: Accepted publickey for testuser1357-2 from 127.0.0.1 port 59302 ssh2: ED25519 SHA256:dNTqOi1m/msvHHEtxr+1rO8VN1zY8nLrraEv5+0rljM
1625s ## PASS test_gssapi_keyex_pubkey_fallback
"""

[Launchpad Janitor (janitor)]

This bug was fixed in the package openssh - 1:8.9p1-3ubuntu0.13

---------------
openssh (1:8.9p1-3ubuntu0.13) jammy-security; urgency=medium

  * SECURITY UPDATE: incorrect DisableForwarding directive behaviour
    - debian/patches/CVE-2025-32728.patch: fix logic error in session.c.
    - CVE-2025-32728

 -- Marc Deslauriers <email address hidden> Fri, 11 Apr 2025 08:05:47 -0400