monitor_read: unpermitted request 48 on server while attempting GSSAPI key exchange

[Niklas Rother]

I'm using openssh 1:8.2p1-4ubuntu0.2 on Ubuntu 20.04.2 LTS (client and server) with the option "GSSAPIKeyExchange=yes", and this causes the connection to fail. The server has GSSAPI (Kerberos authentication) enabled, but is is only used for non-root users (root uses SSH keys).

Client command:

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@server -v -p 2222 -o GSSAPIKeyExchange=yes

Client log:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to compute-test [130.75.80.46] port 2222.
debug1: Connection established.
debug1: identity file /home/rother/.ssh/id_rsa type 0
debug1: identity file /home/rother/.ssh/id_rsa-cert type -1
debug1: identity file /home/rother/.ssh/id_dsa type -1
debug1: identity file /home/rother/.ssh/id_dsa-cert type -1
debug1: identity file /home/rother/.ssh/id_ecdsa type -1
debug1: identity file /home/rother/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/rother/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/rother/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/rother/.ssh/id_ed25519 type -1
debug1: identity file /home/rother/.ssh/id_ed25519-cert type -1
debug1: identity file /home/rother/.ssh/id_ed25519_sk type -1
debug1: identity file /home/rother/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/rother/.ssh/id_xmss type -1
debug1: identity file /home/rother/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to server:2222 as 'root'
debug1: Offering GSSAPI proposal: gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group14-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-gex-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-group14-sha1-eipGX3TCiQSrx573bT1o1Q==
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g==
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none
debug1: Doing group exchange
debug1: Calling gss_init_sec_context
debug1: Delegating credentials
debug1: Received GSSAPI_COMPLETE
debug1: Calling gss_init_sec_context
debug1: Delegating credentials
debug1: Rekey has happened - updating saved versions
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/rother/.ssh/id_rsa RSA SHA256:n/EY/cGjgd/r+7JpuqODxIotHHLsYptGXYx9GlKCWSM agent
debug1: Will attempt key: /home/rother/.ssh/root_id_rsa RSA SHA256:yCLAID9FMILharHmDpCB8wW8eiA+iHa4oQKLODbbzKw agent
debug1: Will attempt key: /home/user/.ssh/id_dsa
debug1: Will attempt key: /home/user/.ssh/id_ecdsa
debug1: Will attempt key: /home/user/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/user/.ssh/id_ed25519
debug1: Will attempt key: /home/user/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/user/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,<email address hidden>,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,<email address hidden>>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Delegating credentials
debug1: Delegating credentials
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
Connection closed by 1.2.3.4 port 2222

Server log:

debug1: sshd version OpenSSH_8.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: private host key #0: ssh-rsa SHA256:REDACTED
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:REDACTED
debug1: private host key #2: ssh-ed25519 SHA256:REDACTED
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2222'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_8.2, OpenSSL 1.1.1f 31 Mar 2020
debug1: private host key #0: ssh-rsa SHA256:REDACTED
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:REDACTED
debug1: private host key #2: ssh-ed25519 SHA256:REDACTED
debug1: inetd sockets after dupping: 3, 3
Connection from 1.2.3.5 port 53724 on 1.2.3.4 port 2222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: permanently_set_uid: 111/65534 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g== [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: <email address hidden> MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: <email address hidden> MAC: <implicit> compression: none [preauth]
debug1: Doing group exchange [preauth]
debug1: Wait SSH2_MSG_GSSAPI_INIT [preauth]
debug1: Received some client credentials
debug1: rekey out after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: Sending SSH2_MSG_EXT_INFO [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey in after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "1.2.3.5"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method gssapi-with-mic [preauth]
debug1: attempt 1 failures 0 [preauth]
Postponed gssapi-with-mic for root from 1.2.3.5 port 53724 ssh2 [preauth]
debug1: Received some client credentials
Failed gssapi-with-mic for root from 1.2.3.5 port 53724 ssh2
debug1: userauth-request for user root service ssh-connection method gssapi-with-mic [preauth]
debug1: attempt 2 failures 1 [preauth]
debug1: userauth-request for user root service ssh-connection method gssapi-keyex [preauth]
debug1: attempt 3 failures 1 [preauth]
monitor_read: unpermitted request 48
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 5525
debug1: audit_event: unhandled event 12

The important line might be "monitor_read: unpermitted request 48"

When disabling GSSAPIKeyExchange=yes, everything works as expected. This bug was discovered using Ansible, which uses "-o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey" for it's ssh connections.

A similar bugs was reported in RHEL 7: https://bugzilla.redhat.com/show_bug.cgi?id=1162620

Please let me know if you need any further information!

[Athos Ribeiro (athos-ribeiro)]

Hello Niklas,

Thank you for taking the time to file a bug report.

While the symptoms experienced here seem similar to the ones reported in https://bugzilla.redhat.com/show_bug.cgi?id=1162620, the patch that fixed the latter is present in the version of the package for which you reported the issue.

Therefore, would you mind providing additional information, such as configuration files? More importantly, we would be interested in a reproducer for the issue.
Can you reproduce it without using ansible?

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/communit

[Niklas Rother (nrother)]

Hello Athos,

thanks for looking into this!

This is reproducible without Ansible, that was just use-case that brought up the issue. I've further narrowed it down to the following setup:

Server:
/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

Client:
ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@compute-test -v -p 2222 -o GSSAPIKeyExchange=yes -F /dev/null

I think this should make it independent from my local config, right? Obviously there is also Kerberos involved, which I would call configured pretty standard in our environment, but I can have a look at that config as well, if this is desired.

The problem will not arise when:
- The client has no valid Kerberos-Key (unset KRB5CCNAME)
- If any of the the GSSAPI* options is missing on client or server
- If the order of "gssapi-with-mic,gssapi-keyex" is switched (!)

[Miriam España Acebal (mirespace)]

Hello Niklas!

You're welcome again. Thanks for adding more information... Good to know we can put Ansible apart from this.

Anyway, we would need more information about the Kerberos configuration you mentioned before as you noticed it is involved because we don't have the complete picture to reproduce the issue. Also sshd configuration is needed for both client and server.

In the time we receive that, I will mark the report as "Incomplete", but thanks a lot for progressing on it.

When you submit new information, please mark the bug as "New" so we can continue with it.

[Sergio Durigan Junior (sergiodj)]

So, I give this a try and attempted to reproduce the issue.

I set up a VM acting as the KDC, and configured sshd in it with the following options:

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
GSSAPIKeyExchange yes

I then configured an LXD container to act as the krb5 client. I created a user "john" both in the KDC and in the client, then was able to verify that kinit was working fine. With that out of the way, I tried to connect via ssh to the KDC:

$ ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex -o GSSAPIKeyExchange=yes krb5.test.lan

The connection worked. I did the RH bug and tried to check if there was anything else I could do, but apparently the bug should have manifested with what I did. I also tried to start sshd by hand using the options you mentioned (plus "-o UsePam=yes"), to no avail. So I'm a bit lost here, and would also appreciate more info.

Thanks.

[Niklas Rother (nrother)]

Thanks Sergio for trying to reproduce this! I'm a bit puzzled, why the bug did not trigger in your case. I'll try to reproduce this in a clean VM as well now. One important thing might be that I tried to login as "root", for which I did not have a Kerberos-Ticket, so a "permission denied" would be expected, unless something like "publickey" is included in PreferredAuthentications.

@Miriam: The SSH-configuration should be irrelevant, because of the options "-f /dev/null" and "-F /dev/null". The Kerberos-config could be part of this. I'll try to reproduce this on a clean machine and post better steps for reproducing afterwards.

Thank you all for helping with this!

[Niklas Rother (nrother)]

Ok, I managed to reproduces this in a clean "ubuntu:latest" docker container. Steps to reproduce are below. During testing, I noticed that I aliased "ssh" to "ssh -K -X", and that "-K" (or equivalently "-o GSSAPIAuthentication=yes") is crucial. This changes the problematic SSH client command to

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@ac3f9944f201 -v -p 2222 -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -F /dev/null

Complete steps to reproduce (container ac3f9944f201 is the server, IP 1.2.3.4 is the IP of the container host; this needs to be adapted):

Server:

podman run -it -p 2222:2222,8888:88 ubuntu

apt update
apt install openssh-server krb5-kdc krb5-admin-server
touch /etc/krb5kdc/kadm5.acl
touch /etc/krb5kdc/kadm5.dict
krb5_newrealm
kadmin.local

addprinc user
addprinc -randkey host/ac3f9944f201
ktadd -k /etc/krb5.keytab host/ac3f9944f201
exit

mkdir /run/sshd
/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

Client:

podman run -it ubuntu

apt update
apt install openssh-client krb5-user
kinit user
echo "1.2.3.4 ac3f9944f201" >> /etc/hosts

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@ac3f9944f201 -v -p 2222 -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -F /dev/null

Notice "monitor_read: unpermitted request 48" on the server, and "Connection closed by 1.2.3.4 port 2222" on the client (instead of the expected "permission denied).

[Athos Ribeiro (athos-ribeiro)]

Hi Niklas,

Thanks for putting in the effort into finding a reproducer for the reported issue.

I could indeed reproduce the issue you have been experiencing. I am attaching a couple scripts to aid others to reproduce the bug (this includes a README file with further instructions).

Interestingly, if you swap the preferred authentications order to read

PreferredAuthentications=gssapi-keyex,gssapi-with-mic

The bug will not manifest itself.

Next, I will verify if other branches at https://github.com/openssh-gsskex/openssh-gsskex are also affected. If this is the case, we should report the issue there.

[Athos Ribeiro (athos-ribeiro)]

The issue is reproducible in the latest published versions of openssh carrying the patches in https://github.com/openssh-gsskex/openssh-gsskex for Ubuntu (impish), Debian (unstable), and Fedora (rawhide).

I filed a bug report in https://github.com/openssh-gsskex/openssh-gsskex/issues/20 to make sure the gsskex patch upstream is aware of this issue.

[Athos Ribeiro (athos-ribeiro)]

Dmitry Belyavskiy proposed a patch for this issue at https://github.com/openssh-gsskex/openssh-gsskex/pull/21.

I created a PPA with the proposed fix at https://launchpad.net/~athos-ribeiro/+archive/ubuntu/openssh-gssapi-fix/+packages and I can confirm it does fix the reproducer proposed in this bug.

Moreover, running the server with

/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -o PasswordAuthentication=yes -o PermitRootLogin=yes

And logging in as root, will prompt for the root password and get you a proper ssh connection.

Finally, I also ran the available openssh dep8 test suite to ensure the patch would not introduce covered regrerssions.

autopkgtest [17:57:18]: @@@@@@@@@@@@@@@@@@@@ summary
regress PASS

Niklas, it would be really nice if you could also test the proposed patch to confirm it does fix the reported issue.

[Niklas Rother (nrother)]

Thanks for the PPA! I currently don't have an impish system at hand, would it be possible to build it for focal as well?

[Athos Ribeiro (athos-ribeiro)]

Hi Niklas,

I just pushed the focal patched package to that same PPA. Note that they are only available for x86_64 and i386. Let me know if you need it for any other platforms.

[Niklas Rother (nrother)]

Thanks for building the packages for focal.

I can confirm that this fixes the problem for me!

[Utkarsh Gupta (utkarsh)]

Woooooooot, thanks for confirming Niklas. Athos, would you mind prepping an MP for this, et al? TIA, good sir! \o/

[Athos Ribeiro (athos-ribeiro)]

Thanks, Niklas!

Utkarsh, Paride: Since this seems to be a low priority issue, I am waiting to see if we get a couple more eyes into https://github.com/openssh-gsskex/openssh-gsskex/pull/21 before adding this one in our delta (this could even go into Debian first and then we can start preparing SRUs). Therefore, I am also removing the server-next tag from this one.

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[Paride Legovini (paride)]

That PR is still pending review, no movements there unfortunately.