moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv

[Stefan Paetow]

When using moonshot-gss-eap-1.0.1-1.el6 on a new CentOS 6 box, I see the following issue:

When used with the newest moonshot-ui package (see Bug 1709316) in a mode that uses .gss_eap_id, RADIUS reports a TLS failure:

eap_ttls: ERROR: TLS Alert read:fatal:unknown CA
eap_ttls: ERROR: TLS_accept: Failed in SSLv3 read client certificate A
eap_ttls: ERROR: Failed in __FUNCTION__ (SSL_read)
eap_ttls: ERROR: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
eap_ttls: ERROR: error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
eap_ttls: ERROR: System call (I/O) error (-1)
eap_ttls: ERROR: TLS receive handshake failed during operation
eap_ttls: ERROR: [eaptls process] = fail
eap: ERROR Failed continuing EAP TTLS (21) session. EAP sub-module failed

When I downgrade to 0.9.5-1, the problem goes away. If there is TLS functionality that attempts to get trust anchors (and fails), perhaps we should update the .gss_eap_id functionality to add a third line that allows a trust anchor?

Additionally, when I downgrade moonshot-ui to avoid Bug 1709316, the moonshot-gss-eap package appears to cause a segv in the ssh process during a call like this:

ssh -Kv <email address hidden>
:
:
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
Segmentation fault (core dumped)

This does not happen when I downgrade to 0.9.5-1.

A virtual machine (Virtual Box 5.1) can be provided that demonstrates this issue.

[Mark Donnelly (meadmaker)]

Stefan, thanks for the bug report. I'll get Dan to look at this after he finishes with But 1709316.

Before that, however, I'm a little confused by your statements here:

> Additionally, when I downgrade moonshot-ui to avoid Bug 1709316
[snip!]
> This does not happen when I downgrade to 0.9.5-1.

It seems like you're saying that the segmentation fault both does and doesn't happen at 0.9.5-1. Could you help me understand what the environment is when you see the bug?

[Stefan Paetow (stefan-paetow)]

> When used with the newest moonshot-ui package (see Bug 1709316) in a mode that uses .gss_eap_id, RADIUS reports a TLS failure:

1. I use .gss_eap_id with the new UI package and the new gss-eap package and the segv does not happen, but I get the RADIUS TLS error (Issue 1)
2. I use .gss_eap_id with the new UI package and the old gss-eap package, the segv does not happen, and neither does the TLS error.

3. I use .gss_eap_id with the old UI package and the new gss-eap package, the segv happens. (Issue 2).
4. I use .gss_eap_id with the old UI package and the old gss-eap package, all is fine.

Seems to be there is an unfortunate Dbus issue or *something* that breaks. Ask Dan for his Centos 6 image that demonstrates Bug 1709316. Then selectively upgrade or downgrade the two packages and observe with SSH. You'll see the problem.

[Dan Breslau (dbreslau)]

Hi, Stefan,

Could you please check what version of Freeradius is in use? The "tlsv1" part of "tlsv1 alert unknown ca" makes me think it's a version older than 3.0.10 (IIRC); those versions didn't play nicely with TLSv1.1 or TLSv1.2.

Also, we apparently failed to put a necessary dependency on the mech_eap library: moonshot-gss-eap 1.0.1 definitely needs to run with a Moonshot UI greater than (I think) 0.9.6. However, I don't think there was a released Centos 6 package for the UI until 1.0.5, which of course is broken. The good news that we're just about ready to go live with 1.0.6...

[Dan Breslau (dbreslau)]

Correction to above: I meant to write that older versions of Freeradius don't play nicely with TLSv1.2. I'm not sure whether it works with TLSv1.1. But in any case, the newer mech_eap library insists on using TLSv1.2.

[Stefan Paetow (stefan-paetow)]

The system in question (a test system for our macOS team) uses FR 3.0.15. Default setup with only our required changes at this point. It was set up from the CentOS 6.9 LiveDVD, updated to the latest of *everything*. I will post a link to the image (privately) if that's helpful.

[Dan Breslau (dbreslau)]

Is this the same image that you just emailed me a link to on another thread?

It would also be very helpful if you could include details about the command[s] you're running when you see this problem. I gather that ssh is one of those commands. I'm guessing that that's a moonshot-enabled ssh; and if so, what version?

[Stefan Paetow (stefan-paetow)]

Yes.

*Everything* is the latest, including the latest MS package for OpenSSH (other than the GSS-EAP and MS-UI packages - for obvious reasons). The bash history in the 'sysuser' user has everything you need... The command is ssh -Kv <email address hidden> (the local machine name).

Stefan Paetow
Moonshot Industry & Research Liaison Coordinator

t: [phone]
gpg: 0x3FCE5142
xmpp: <email address hidden>
skype: stefan.paetow.janet

jisc.ac.uk

Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.

From: <<email address hidden><mailto:<email address hidden>>> on behalf of Dan Breslau <<email address hidden><mailto:<email address hidden>>>
Reply-To: Bug 1709337 <<email address hidden><mailto:<email address hidden>>>
Date: Tuesday, 8 August 2017 at 23:17
To: Stefan Paetow <<email address hidden><mailto:<email address hidden>>>
Subject: [Bug 1709337] Re: moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv

Is this the same image that you just emailed me a link to on another
thread?

It would also be very helpful if you could include details about the
command[s] you're running when you see this problem. I gather that ssh
is one of those commands. I'm guessing that that's a moonshot-enabled
ssh; and if so, what version?

--
You received this bug notification because you are a member of Moonshot
Drivers, which is subscribed to Project Moonshot.
Matching subscriptions: Moonshot Drivers
https://bugs.launchpad.net/bugs/1709337

Title:
  moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv

Status in Project Moonshot:
  New

Bug description:
  When using moonshot-gss-eap-1.0.1-1.el6 on a new CentOS 6 box, I see
  the following issue:

  When used with the newest moonshot-ui package (see Bug 1709316) in a
  mode that uses .gss_eap_id, RADIUS reports a TLS failure:

  eap_ttls: ERROR: TLS Alert read:fatal:unknown CA
  eap_ttls: ERROR: TLS_accept: Failed in SSLv3 read client certificate A
  eap_ttls: ERROR: Failed in __FUNCTION__ (SSL_read)
  eap_ttls: ERROR: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
  eap_ttls: ERROR: error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
  eap_ttls: ERROR: System call (I/O) error (-1)
  eap_ttls: ERROR: TLS receive handshake failed during operation
  eap_ttls: ERROR: [eaptls process] = fail
  eap: ERROR Failed continuing EAP TTLS (21) session. EAP sub-module failed

  When I downgrade to 0.9.5-1, the problem goes away. If there is TLS
  functionality that attempts to get trust anchors (and fails), perhaps
  we should update the .gss_eap_id functionality to add a third line
  that allows a trust anchor?

  Additionally, when I downgrade moonshot-ui to avoid Bug 1709316, the
  moonshot-gss-eap package appears to cause a segv in the ssh process
  during a call like this:

  ssh -Kv <email address hidden><mailto:<email address hidden>>
  :
  :
  debug1: Next authentication method: gssapi-keyex
  debug1...

[Dan Breslau (dbreslau)]

Thanks, Stefan. Unfortunately, I'm not able to get the VM to boot up.
I'm running VB 5.1.22. I've found that snapshots can be problematic even
between VB point releases. Would it be a problem for you to delete the
snapshot, and then export the VM as an appliance?

That said -- I think I have managed to reproduce the issue using
gss-client and gss-server, so if this *would* be a problem, no worries
(yet :-) .

Thanks,

Dan

On 8/9/2017 3:37 AM, Stefan Paetow wrote:
> Yes.
>
> *Everything* is the latest, including the latest MS package for OpenSSH
> (other than the GSS-EAP and MS-UI packages - for obvious reasons). The
> bash history in the 'sysuser' user has everything you need... The
> command is ssh -Kv <email address hidden> (the local machine
> name).
>
> Stefan Paetow
> Moonshot Industry & Research Liaison Coordinator
>
> t: [phone]
> gpg: 0x3FCE5142
> xmpp: <email address hidden>
> skype: stefan.paetow.janet
>
> jisc.ac.uk
>
> Jisc is a registered charity (number 1149740) and a company limited by
> guarantee which is registered in England under Company No. 5747339, VAT
> No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower
> Hill, Bristol, BS2 0JA. T 0203 697 5800.
>
> From: <<email address hidden><mailto:<email address hidden>>> on behalf of Dan Breslau <<email address hidden><mailto:<email address hidden>>>
> Reply-To: Bug 1709337 <<email address hidden><mailto:<email address hidden>>>
> Date: Tuesday, 8 August 2017 at 23:17
> To: Stefan Paetow <<email address hidden><mailto:<email address hidden>>>
> Subject: [Bug 1709337] Re: moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv
>
> Is this the same image that you just emailed me a link to on another
> thread?
>
> It would also be very helpful if you could include details about the
> command[s] you're running when you see this problem. I gather that ssh
> is one of those commands. I'm guessing that that's a moonshot-enabled
> ssh; and if so, what version?
>
> --
> You received this bug notification because you are a member of Moonshot
> Drivers, which is subscribed to Project Moonshot.
> Matching subscriptions: Moonshot Drivers
> https://bugs.launchpad.net/bugs/1709337
>
> Title:
> moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv
>
> Status in Project Moonshot:
> New
>
> Bug description:
> When using moonshot-gss-eap-1.0.1-1.el6 on a new CentOS 6 box, I see
> the following issue:
>
> When used with the newest moonshot-ui package (see Bug 1709316) in a
> mode that uses .gss_eap_id, RADIUS reports a TLS failure:
>
> eap_ttls: ERROR: TLS Alert read:fatal:unknown CA
> eap_ttls: ERROR: TLS_accept: Failed in SSLv3 read client certificate A
> eap_ttls: ERROR: Failed in __FUNCTION__ (SSL_read)
> eap_ttls: ERROR: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
> eap_ttls: ERROR: error:140940E5:SSL routines:SSL3_READ_BYTES:ssl handshake failure
> eap_ttls: ERROR: System call (I/O) error (-1)
> eap_ttls: ERROR: TLS receive handshake failed during operation
> eap_ttls: ERROR: [eaptls process] = fail
> eap: ERROR Failed continuing EA...

[Dan Breslau (dbreslau)]

No, scratch that: I may have received the same "unknown CA" message in
the freeradius logs, but that was apparently for an auth request that
succeeded anyway. (In which case, "unknown CA" may have been more of a
warning than an error -- at least in my case.) I'll look into it more
when I get into the office.

-- Dan

On 8/9/2017 7:54 AM, Dan Breslau wrote:
> Thanks, Stefan. Unfortunately, I'm not able to get the VM to boot up.
> I'm running VB 5.1.22. I've found that snapshots can be problematic
> even between VB point releases. Would it be a problem for you to
> delete the snapshot, and then export the VM as an appliance?
>
> That said -- I think I have managed to reproduce the issue using
> gss-client and gss-server, so if this *would* be a problem, no worries
> (yet :-) .
>
> Thanks,
>
> Dan
>
> On 8/9/2017 3:37 AM, Stefan Paetow wrote:
>> Yes.
>>
>> *Everything* is the latest, including the latest MS package for OpenSSH
>> (other than the GSS-EAP and MS-UI packages - for obvious reasons). The
>> bash history in the 'sysuser' user has everything you need... The
>> command is ssh -<email address hidden> (the local machine
>> name).
>>
>> Stefan Paetow
>> Moonshot Industry & Research Liaison Coordinator
>>
>> t: [phone]
>> gpg: 0x3FCE5142
>> xmpp:<email address hidden>
>> skype: stefan.paetow.janet
>>
>> jisc.ac.uk
>>
>> Jisc is a registered charity (number 1149740) and a company limited by
>> guarantee which is registered in England under Company No. 5747339, VAT
>> No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower
>> Hill, Bristol, BS2 0JA. T 0203 697 5800.
>>
>> From: <<email address hidden><mailto:<email address hidden>>> on behalf of Dan Breslau <<email address hidden><mailto:<email address hidden>>>
>> Reply-To: Bug 1709337 <<email address hidden><mailto:<email address hidden>>>
>> Date: Tuesday, 8 August 2017 at 23:17
>> To: Stefan Paetow <<email address hidden><mailto:<email address hidden>>>
>> Subject: [Bug 1709337] Re: moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv
>>
>> Is this the same image that you just emailed me a link to on another
>> thread?
>>
>> It would also be very helpful if you could include details about the
>> command[s] you're running when you see this problem. I gather that ssh
>> is one of those commands. I'm guessing that that's a moonshot-enabled
>> ssh; and if so, what version?
>>
>> --
>> You received this bug notification because you are a member of Moonshot
>> Drivers, which is subscribed to Project Moonshot.
>> Matching subscriptions: Moonshot Drivers
>> https://bugs.launchpad.net/bugs/1709337
>>
>> Title:
>> moonshot-gss-eap-1.0.1-1.el6 causes RADIUS TLS error and SSH segv
>>
>> Status in Project Moonshot:
>> New
>>
>> Bug description:
>> When using moonshot-gss-eap-1.0.1-1.el6 on a new CentOS 6 box, I see
>> the following issue:
>>
>> When used with the newest moonshot-ui package (see Bug 1709316) in a
>> mode that uses .gss_eap_id, RADIUS reports a TLS failure:
>>
>> eap_ttls: ERROR: TLS Alertread:fatal:unknown CA
>> eap_ttls: ERROR: TLS_accept: Failed in SSLv3 read client ce...

[Dan Breslau (dbreslau)]

This bug was introduced in rev 344d7f981a4d7d1ef7d8f8d7645aa9c1d153b6cf of www.project-moonshot.org:/srv/git/mech_eap.git (aka moonshot/moonshot). I've reproduced it on Centos7 and Debian. The common factor is that it happens when we fall back to using the .gss-eap-id file for identities.

Note that running the ID Selector in headless mode provides more functionality than that offered by using the .gss-eap-id file.