Negative interaction with Kerberos

[Adam Bishop]

0. Moonshot enable client
1. Enroll client in kerberos realm A (DEV.JA.NET)
2. kdestroy
3. Obtain ticket for kerberos realm B (VIRT.JA.NET)
4. Attempt to connect to server in kerberos realm A
5. OpenSSH client encounters a problem runs its exception handler and exits

This is very much an edge case, and I can provide a system running CentOS that replicates this problem.

I'm don't think this is a moonshot problem per se, but it would be useful to identify the root cause, and potentially supply a patch upstream.

EDIT: Should have noted this doesn't seem to occur when the moonshot mechanism is absent.

[Adam Bishop (adam-omega)]

#0 0x00007ffff5b66f38 in _exit (status=255)
   at ../sysdeps/unix/sysv/linux/_exit.c:33
       resultvar = 18446744073709551578
#1 0x00007ffff7fa2f5c in cleanup_exit (i=255) at clientloop.c:2183
No locals.
#2 0x00007ffff7fc1e1f in fatal (fmt=<value optimized out>) at fatal.c:44
       args = {{gp_offset = 8, fp_offset = 48,
           overflow_arg_area = 0x7fffffffc4f0,
           reg_save_area = 0x7fffffffc430}}
#3 0x00007ffff7fdc036 in kexgss_client (kex=0x7ffff824db40) at kexgssc.c:142
       send_tok = {length = 0, value = 0x0}
       recv_tok = {length = 186, value = 0x7ffff825a3b0}
       gssbuf = {length = 140737356520720, value = 0x1}
       msg_tok = {length = 140737356553360, value = 0x7ffff8258890}
       token_ptr = 0x7fffffffc600
       ctxt = 0x7ffff8257210
       maj_status = 851968
       min_status = <value optimized out>
       ret_flags = 0
       klen = <value optimized out>
       kout = <value optimized out>
       slen = 0
       hashlen = <value optimized out>
       strlen = 186
       dh = 0x7ffff8258770
       dh_server_pub = 0x7ffff8257500
       shared_secret = 0x0
       p = <value optimized out>
       g = <value optimized out>
       kbuf = <value optimized out>
       hash = <value optimized out>
       serverhostkey = 0x0
       empty = 0x7ffff7ff1e72 ""
       msg = <value optimized out>
       type = 31
       first = 0
       nbits = 0
       __func__ = "kexgss_client"
#4 0x00007ffff7fd1bd8 in kex_kexinit_finish (type=<value optimized out>,
   seq=<value optimized out>, ctxt=0x7ffff824db40) at kex.c:320
No locals.
#5 kex_input_kexinit (type=<value optimized out>, seq=<value optimized out>,
   ctxt=0x7ffff824db40) at kex.c:290
       ptr = <value optimized out>
       i = <value optimized out>
       dlen = 1308
       kex = 0x7ffff824db40
#6 0x00007ffff7fd0c83 in dispatch_run (mode=0, done=0x7ffff824dba8,
   ctxt=0x7ffff824db40) at dispatch.c:98
       type = <value optimized out>
       seqnr = 0
#7 0x00007ffff7fadf58 in ssh_kex2 (host=<value optimized out>,
   hostaddr=<value optimized out>) at sshconnect2.c:230
       kex = 0x7ffff824db40
       orig = <value optimized out>
       gss = <value optimized out>
       gss_host = 0x7ffff822a290 "ms-ssh-rp.dev.ja.net"
#8 0x00007ffff7fa96eb in ssh_login (sensitive=0x7ffff82033a0,
   orighost=<value optimized out>, hostaddr=0x7ffff82033c0,
   pw=<value optimized out>, timeout_ms=-1000) at sshconnect.c:1137
       host = 0x7ffff822a290 "ms-ssh-rp.dev.ja.net"
       cp = <value optimized out>
       server_user = 0x7ffff82251f0 "adamb"
       local_user = 0x7ffff8225260 "adamb"
#9 0x00007ffff7f9f7de in main (ac=<value optimized out>,
   av=<value optimized out>) at ssh.c:904
       i = <value optimized out>
       r = <value optimized out>
       opt = <value optimized out>
       exit_status = <value optimized out>
       use_syslog = -131968416
       p = <value optimized out>
       cp = <value optimized out>
       line = <value optimized out>
       argv0 = <value optimized out>
       buf = "/home/DEV/adamb/.ssh\000config\000\000\000\000\000\300\320\377\377\377\177\000\000\037\310\345\365\377\177\000\000\310\064\370\367\377\177\000\000\000\000 \2...

[Adam Bishop (adam-omega)]

[adamb@ms-ssh-rp ~]$ ssh -vv ms-ssh-rp.dev.ja.net
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ms-ssh-rp.dev.ja.net [212.219.210.74] port 22.
debug1: Connection established.
debug1: identity file /home/DEV/adamb/.ssh/identity type -1
debug1: identity file /home/DEV/adamb/.ssh/identity-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_rsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_rsa-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_dsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_dsa-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_ecdsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 4 setting O_NONBLOCK
debug1: Unspecified GSS failure. Minor code may provide more information
Server <email address hidden> not found in Kerberos database

debug1: Unspecified GSS failure. Minor code may provide more information
Server <email address hidden> not found in Kerberos database

debug1: Unspecified GSS failure. Minor code may provide more information

debug1: Unspecified GSS failure. Minor code may provide more information
Missing default password or other credentials

debug1: Unspecified GSS failure. Minor code may provide more information
Missing default password or other credentials

debug1: Offering GSSAPI proposal: gss-group1-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-group14-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-gex-sha1-eipGX3TCiQSrx573bT1o1Q==
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: gss-group1-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-group14-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-gex-sha1-eipGX3TCiQSrx573bT1o1Q==,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,ssh-rsa,ssh-dss,null
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: fi...

[Sam Hartman (hartmans)]

Can I get you to try turning off GssapiKeyExchange but leaving on
GssapiAuthentication?

An unfortunate side effect of the openssh protocol is that if you select
a particular GSS mechanism but cannot complete it, the ssh server has no
choice but to fail the key exchange.
That is, the server and client get exactly one shot to make key exchange
work.

So if one side thinks Kerberos will work and that turns out to not be
the case, you get into this situation.

It may be that the ssh gssapi patches are prematurely committing to key
exchange.

I wonder if you're jumping down the path of iakerb or some other
Kerberos-derived mechanism or something.
What's happening is that the client's initial gss_init_sec_context call
is succeeding, so the client commits to that key exchange mechanism, but
a couple steps in, the client discovers it cannot get a ticket.
For normal Kerberos, that happens in the first round trip.

This might kind of be our fault.
Our patches add support for arbitrary mechanisms based on their
mechanism attributes rather than having a hard-coded list.
In this instance, we may be regretting that.
I'm not entirely sure what a good solution is here.

[Adam Bishop (adam-omega)]

Disabling GSSAPI Key Exchange in the client config makes it proceed as expected, so that is a workaround.

Deleting the moonshot mech file doesn't change the behaviour, so it would seem that the GSS patches are at fault, rather than the presence of our mechanism directly.

[adamb@ms-ssh-rp ~]$ ssh -vv -l "" ms-ssh-rp.dev.ja.net
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ms-ssh-rp.dev.ja.net [212.219.210.74] port 22.
debug1: Connection established.
debug1: identity file /home/DEV/adamb/.ssh/identity type -1
debug1: identity file /home/DEV/adamb/.ssh/identity-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_rsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_rsa-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_dsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_dsa-cert type -1
debug1: identity file /home/DEV/adamb/.ssh/id_ecdsa type -1
debug1: identity file /home/DEV/adamb/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: <email address hidden>,<email address hidden>,<email address hidden>,<email address hidden>,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,<email address hidden>
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,<email address hidden>,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,<email address hidden>,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit: none,<email address hidden>,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group14-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group1-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-group14-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-gex-sha1-eipGX3TCiQSrx573bT1o1Q==,gss-group1-sha1-JHo1wNMhky/7DkDu4d6xCA==,gss-group14-sha1-JHo1wNMhky/7DkDu4d6xCA==,gss-gex-sha1-JHo1wNMhky/7DkDu4d6xCA==,gss-group1-sha1-dEYdZI86nhHqawDlBMslQw==,gss-group14-sha1-dEYdZI86nhHqawDlBMslQw==,gss-gex-sha1-dEYdZI86nhHqawDlBMslQw==,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sh...

[Stefan Paetow (stefan-paetow)]

Confirmed at Bath, it works without GSSAPIKeyExchange

[Stefan Paetow (stefan-paetow)]

So, this is a solution for the current problem, but we may want to consider fixing the problem correctly/appropriately.