SSL problem with PHP 5.6.7&.8 and Horde_Imap_Client

[RalfBecker]

I created a simple example code reproducing the problem Horde or EGroupware:

<?php
$timeout = 30;
$context = array(
    'ssl' => array(
        'verify_peer' => false,
        'verify_peer_name' => false
    )
);
if (($fp = stream_socket_client(
        $conn='ssl://imap.ikmj.com:993',
        $errno,
        $errstr,
        $timeout,
        STREAM_CLIENT_CONNECT,
        stream_context_create($context)
    )))
{
    echo "Connected to $conn\n";
    echo fgets($fp);

    echo "1 CAPABILITY\n";
    fwrite($fp, "1 CAPABILITY\n");
    echo fgets($fp);
    echo "2 LOGOUT\n";
    fwrite($fp, "2 LOGOUT\n");
}
else var_dump($errno, $errstr);

Running it on various IUS PHP versions gives following results:
[root@lighttpd ~]# php -v
PHP 5.4.39 (cli) (built: Mar 20 2015 08:10:43)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
[root@lighttpd ~]# php imaps_php56.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS
UNSELECT XLIST AUTH=PLAIN AUTH=LOGIN
2 LOGOUT

[root@fpm60 ~]# php -v
PHP 5.5.23 (cli) (built: Mar 20 2015 08:29:54)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend
Technologies
[root@fpm60 ~]# php imaps_php56.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS
UNSELECT XLIST AUTH=PLAIN AUTH=LOGIN
2 LOGOUT

[dev@coreos-control ~]$ php -v
PHP 5.6.7 (cli) (built: Mar 25 2015 13:08:46)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
[dev@coreos-control ~]$ php imaps_php56.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
^C
Hangs until I hit ^C.

[dev@coreos-control ~]$ php -v
PHP 5.6.8 (cli) (built: Apr 16 2015 20:02:05)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
[dev@coreos-control ~]$ php imaps_php56.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
^C
Hangs until I hit ^C.

While I first suspected a further case of SSL timeout bug we had in 5.5 and 5.6 a couple of packages back, I also talked to Remi Collet who reported the previous problem to php internals ML and maintains PHP in Fedora.
Hi could NOT reproduce it on various PHP & Fedora versions:

$ php56 /tmp/foo.php
Running PHP 5.6.8
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS
UNSELECT XLIST AUTH=PLAIN AUTH=LOGIN
2 LOGOUT

Hi just run more tests
- Fedora 21 and openssl 1.0.1k
- RHEL-6 and RHEL-7, openssl 1.0.1e
- PHP 5.4.40, 5.5.24, 5.6.8 and 7.0.0-dev (20150418)

So it appears to me that there must be some difference in patches or spec file or IUS and Fedora which causes the problem with IUS PHP 5.6 packages.

Ralf

[bharper (bharper)]

Hello Ralf,

Thanks for taking the time to submit this bug. I am able to reproduce this issue with php56u packages. We will look into this and hopefully have something in the testing repos soon.

-Ben

[RalfBecker (ralfbecker)]

Hi Ben,

just got following reply from Daniel Lowrey <email address hidden> PHP openssl maintainer:

Hi folks,

This issue was first reported after the release of 5.6.7. It has been corrected upstream but this was done after 5.6.8 had already been tagged and so the fix is not present in that release. Current 5.6 snapshots resolve the issue and the fix will be present in the 5.6.9 release.

I asked him to point out the commit, but got no reply so far. Maybe Remi added that patch to his 5.6.8 builds ...

I'll keep you posted. Maybe it's a straight forward patch you can include in your 5.6.8 package in testing.

Ralf

[RalfBecker (ralfbecker)]

Remi pointed me to this php bug and commits mentioned there:

https://bugs.php.net/bug.php?id=69402

Ralf

[bharper (bharper)]

I have a slight clarification. In my testing with php56u and your test code, I do get a time out:

$ time php ssl_bug.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
2 LOGOUT

real 0m30.737s
user 0m0.013s
sys 0m0.009s

I assume you will reach the time out if you wait long enough.

I have also discovered that your test script works fine when using a different IMAP server:

$ php ssl_bug_different_server.php
Connected to ssl://<redacted>:993
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready.
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN
2 LOGOUT

[RalfBecker (ralfbecker)]

Yep, I can confirm there are other IMAP server my code works with. Forgot to mention that.

If you use openssl s_client to send the command to that IMAP server it works too:

RalfsMac:egroupware ralf$ openssl s_client -host imap.ikmj.com -port 993
CONNECTED(00000003)
depth=1 C = BE, O = GlobalSign nv-sa, CN = AlphaSSL CA - SHA256 - G2
verify error:num=20:unable to get local issuer certificate
---
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS UNSELECT XLIST AUTH=PLAIN AUTH=LOGIN
1 OK Completed
2 LOGOUT
* BYE IMAP4rev1 logging out
2 OK Goodbye
closed

Ralf

[bharper (bharper)]

Changing the script is use a tcp connection on port 143 works as expected:

$ php ssl_bug.php
Connected to tcp://imap.ikmj.com:143
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS UNSELECT XLIST STARTTLS AUTH=PLAIN AUTH=LOGIN
2 LOGOUT

It appears to be more of a ssl issue than an imap issue.

I also noticed that the ssl certificate does not include the domain name of imap.ikmj.com. Changing the script to use ikmj-serwer.home.pl results in the same time out.

[RalfBecker (ralfbecker)]

Got now the response from Daniel:

No problem. The issue was addressed by http://git.php.net/?p=php-src.git;a=blobdiff;f=ext/openssl/xp_ssl.c;h=e24d32cd45a8ff429e852eb6f1a8f4e7b437ac8f;hp=adef9120d614985d9593dc1411fab262f1a05cbc;hb=81f356b4aea79cd8f86e16096c7ccfbc455bfc3c;hpb=ed7e6d4758b8a8999b5e29e144e394782d473654

There were some long-standing bugs (like 7+ years old) related to stream timeouts not being observed on encrypted streams that were fixed in 5.5.23 and 5.6.7 ... however there were a couple of hiccups (like this one) from those fixes. As far as I can tell these have all been corrected upstream and those changes will be reflected in the next set of bugfix releases.

That's the commit Remi was mentioning too.

Do you consider integration that as patch for 5.6.8 packages in testing?
It looks straight forward to me.
I would give it a test tomorrow, if you can make it available via testing repo.

Ralf

[bharper (bharper)]

Thanks for all the information. Do you this as a single issue or two? It seems that there is one issue in regards to not getting the CAPABILITY data and then another one about the time out. Is this your understanding?

[bharper (bharper)]

I did a test build with that patch and it made things worse. Not only did it not give the CAPABILITY data, but took twice as long to time out:

$ time php ssl_bug.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
2 LOGOUT

real 1m1.536s
user 0m0.012s
sys 0m0.009s

[RalfBecker (ralfbecker)]

It's the not getting of CAPABILITY, data is there and therefore it should not block.

Current situation is it blocks AND even after timeout does not return any data :-(

If the patch makes it worse, it makes of cause no sense to include it in your 5.6.8 builds. I will forward that information to that openssl guy from PHP. So we might get it fixed for 5.6.9.

Thanks for your time :-)

Ralf

[bharper (bharper)]

Hey Ralf,

Thanks for the clarification.

I got looking at the differences in the Fedora spec file and ours. For the most part they are very similar, as we use theirs as a starting point. They use a new compile option, '--with-system-ciphers'. Here is the change log entry:

* Fri Oct 03 2014 Remi Collet <email address hidden> 5.6.1-1
- Update to PHP 5.6.1
  http://php.net/releases/5_6_1.php
- use default system cipher list by Fedora policy
  http://fedoraproject.org/wiki/Changes/CryptoPolicy

If I blacklist the cipher (RC4-SHA) used for imap.ikmj.com within/etc/crypto-policies/back-ends/openssl.config, I get the following:

$ time php ssl_test.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
2 LOGOUT

real 0m30.900s
user 0m0.031s
sys 0m0.023s

Return the configuration back to stock, I get the following:

$ time php ssl_test.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj-serwer.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
* CAPABILITY IMAP4rev1 LITERAL+ CHILDREN I18NLEVEL=1 IDLE SORT UIDPLUS UNSELECT XLIST AUTH=PLAIN AUTH=LOGIN
2 LOGOUT

real 0m0.996s
user 0m0.031s
sys 0m0.017s

So I am wondering if the cipher is to blame for our packages. Seeing that the crypto-policies package is not in RHEL yet, I don't think using the '--with-system-ciphers' option is a good idea. If the '--with-system-ciphers' is not used, a hard coded list will be used. The IUS packages do not change that list and I would be hesitant to make changes to it.

Are you talking with the devs on a mailing list?

[RalfBecker (ralfbecker)]

Hi Daniel,

thanks for looking into it :-)

I'm communicating with Ben from IUS via Launchpad:

    https://bugs.launchpad.net/bugs/1446633

He reported some interesting findings, why my script works for Remi's
packages in Fedora, but not for theirs or Mac Ports.

I'll try to add Launchpad bug email address to this conversation.

Ralf

Am 23.04.15 um 06:11 schrieb Daniel Lowrey:
> It turns out I didn't look closely enough at the issue originally and
> thought it was the same as a separate non-blocking timeout issue. This
> meant the commit I first linked as the "fix" actually didn't address
> your problem.
>
> I spent the last couple of hours trying to isolate the source of the
> problem tonight and I think I'm closing in on it ... There's
> definitely something strange going on. I won't give you the full
> rundown of things right now but I wanted to update you before calling
> it a night since I said earlier that I would.
>
> I'll continue working on this tomorrow and will update you as I know more.
>
>
> On Wed, Apr 22, 2015 at 11:45 AM Daniel Lowrey <<email address hidden>
> <mailto:<email address hidden>>> wrote:
>
> Sure. I'm out of the office for the next couple of hours. I'll try
> out the attached repro script later and get back to you.
>
> On Apr 22, 2015 11:30 AM, "Ralf Becker" <<email address hidden>
> <mailto:<email address hidden>>> wrote:
>
> Hi Daniel,
>
> Ben Harper from IUS did a test-build of their PHP 5.6.8 package
> including your commit as patch and reported it makes things
> worse :-(
>
> Can you run my example script against the imap server
> specified there
> with your current sources?
> Just to be sure it is really fixed and not just a similar issue.
>
> Ralf
>
> Am 21.04.15 um 19:18 schrieb Daniel Lowrey:
> > No problem. The issue was addressed by
> http://git.php.net/?p=php-src.git;a=blobdiff;f=ext/openssl/xp_ssl.c;h=e24d32cd45a8ff429e852eb6f1a8f4e7b437ac8f;hp=adef9120d614985d9593dc1411fab262f1a05cbc;hb=81f356b4aea79cd8f86e16096c7ccfbc455bfc3c;hpb=ed7e6d4758b8a8999b5e29e144e394782d473654
> >
> > There were some long-standing bugs (like 7+ years old)
> related to
> stream timeouts not being observed on encrypted streams that
> were fixed
> in 5.5.23 and 5.6.7 ... however there were a couple of hiccups
> (like
> this one) from those fixes. As far as I can tell these have
> all been
> corrected upstream and those changes will be reflected in the
> next set
> of bugfix releases.
> >
> >
> >
> > On Tue, Apr 21, 2015 at 11:34 AM Remi Collet
> <<email address hidden> <mailto:<email address hidden>>
> <mailto:<email address hidden>
> <mailto:<email address hidden>>>> wrote:
> >
> > Le 21/04/2015 16:43, Ralf Becker a écrit :
> > > Can you name the commit with the fix.
> >
> > I think :
> >
> >
> http://git.php.net/?p=php-src.git;a=commitdiff;h=601d60a978b9e053ab8e6dc
> > 0f12ff850fc642ced
> >
> <http://git....

[RalfBecker (ralfbecker)]

Nope it's still a private mail exchange with a long CC list, I'm happy to add you, if I had your email address.

It seems strange that enabling or disabling a cipher has any other effect then not establishing the communication at all.

Ralf

[bharper (bharper)]

Hey Ralf,

Thanks for the additional information. It sounds like Daniel is deep into his investigation. I think it might be best to let the expert work his magic. If there are questions that I need to address, please feel free to CC the email address in my profile.

-Ben

[RalfBecker (ralfbecker)]

Hi Daniel,

any news on the issue, as 5.6.9 RC1 was just released.

Ralf

Am 23.04.15 um 07:47 schrieb Ralf Becker:
> Hi Daniel,
>
> thanks for looking into it :-)
>
> I'm communicating with Ben from IUS via Launchpad:
>
> https://bugs.launchpad.net/bugs/1446633
>
> He reported some interesting findings, why my script works for Remi's
> packages in Fedora, but not for theirs or Mac Ports.
>
> I'll try to add Launchpad bug email address to this conversation.
>
> Ralf
>
> Am 23.04.15 um 06:11 schrieb Daniel Lowrey:
>> It turns out I didn't look closely enough at the issue originally and
>> thought it was the same as a separate non-blocking timeout issue. This
>> meant the commit I first linked as the "fix" actually didn't address
>> your problem.
>>
>> I spent the last couple of hours trying to isolate the source of the
>> problem tonight and I think I'm closing in on it ... There's
>> definitely something strange going on. I won't give you the full
>> rundown of things right now but I wanted to update you before calling
>> it a night since I said earlier that I would.
>>
>> I'll continue working on this tomorrow and will update you as I know more.
>>
>>
>> On Wed, Apr 22, 2015 at 11:45 AM Daniel Lowrey <<email address hidden>
>> <mailto:<email address hidden>>> wrote:
>>
>> Sure. I'm out of the office for the next couple of hours. I'll try
>> out the attached repro script later and get back to you.
>>
>> On Apr 22, 2015 11:30 AM, "Ralf Becker" <<email address hidden>
>> <mailto:<email address hidden>>> wrote:
>>
>> Hi Daniel,
>>
>> Ben Harper from IUS did a test-build of their PHP 5.6.8 package
>> including your commit as patch and reported it makes things
>> worse :-(
>>
>> Can you run my example script against the imap server
>> specified there
>> with your current sources?
>> Just to be sure it is really fixed and not just a similar issue.
>>
>> Ralf
>>
>> Am 21.04.15 um 19:18 schrieb Daniel Lowrey:
>> > No problem. The issue was addressed by
>> http://git.php.net/?p=php-src.git;a=blobdiff;f=ext/openssl/xp_ssl.c;h=e24d32cd45a8ff429e852eb6f1a8f4e7b437ac8f;hp=adef9120d614985d9593dc1411fab262f1a05cbc;hb=81f356b4aea79cd8f86e16096c7ccfbc455bfc3c;hpb=ed7e6d4758b8a8999b5e29e144e394782d473654
>> >
>> > There were some long-standing bugs (like 7+ years old)
>> related to
>> stream timeouts not being observed on encrypted streams that
>> were fixed
>> in 5.5.23 and 5.6.7 ... however there were a couple of hiccups
>> (like
>> this one) from those fixes. As far as I can tell these have
>> all been
>> corrected upstream and those changes will be reflected in the
>> next set
>> of bugfix releases.
>> >
>> >
>> >
>> > On Tue, Apr 21, 2015 at 11:34 AM Remi Collet
>> <<email address hidden> <mailto:<email address hidden>>
>> <mailto:<email address hidden>
>> <mailto:<email address hidden>>>> wrote:
>> >
>> > Le 21/04/2015 16:43, Ralf Becker a écrit :
>> > > Can you name the commit with the fix.
>>...

[bharper (bharper)]

Hey Ralf,

Just wondering if you have any additional information.

-Ben

[RalfBecker (ralfbecker)]

Nope, haven't heard anything :-(

I hope for 5.6.9

Ralf
--
Ralf Becker
Director Software Development

Stylite AG

Morschheimer Strasse 15 | Tel. [phone]
D-67292 Kirchheimbolanden | Fax. [phone]

Email: <email address hidden>

www.stylite.de | www.egroupware.org

Managing Directors: Andre Keller | Ralf Becker | Gudrun Mueller
Chairman of the supervisory board: Prof. Dr. Birger Leon Kropshofer

VAT DE214280951 | Registered HRB 31158 Kaiserslautern Germany

> Am 12.05.2015 um 22:44 schrieb bharper <email address hidden>:
>
> Hey Ralf,
>
> Just wondering if you have any additional information.
>
> -Ben
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1446633
>
> Title:
> SSL problem with PHP 5.6.7&.8 and Horde_Imap_Client
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ius/+bug/1446633/+subscriptions

[RalfBecker (ralfbecker)]

Looks like the problem is not fixed with php 5.6.9 :-(

[dev@coreos-control ~]$ php -v
PHP 5.6.9 (cli) (built: May 15 2015 13:54:19)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
[dev@coreos-control ~]$ php imaps_php56.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
^C

Ralf

[bharper (bharper)]

Hello Ralf,

I am also able to recreate this issue with php56u:

$ php -v
PHP 5.6.9 (cli) (built: May 15 2015 13:54:40)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies

$ time php ssl_bug.php
Connected to ssl://imap.ikmj.com:993
* OK ikmj.home.pl IdeaImapServer v0.80 ready
1 CAPABILITY
2 LOGOUT

real 1m2.457s
user 0m0.012s
sys 0m0.008s

I still think this issue is tied with the cipher used in your certificate. Your certificate is using RC4-SHA and it appears that PHP is disabling all RC4 ciphers[0]. I have tested with a few different IMAP server that used ciphers on that list and did not run into any issues. You might want to try a test using a certificate with a ciphers on that list.

-Ben

[0] https://github.com/php/php-src/blob/master/ext/openssl/php_openssl.h#L39-L45

[bharper (bharper)]

Just a quick clarification. I think a self-signed certificate would just fine for testing a different cipher.

[bharper (bharper)]

We are moving bug/ticket/issue tracking from Launchpad to GitHub[0]. All stale tickets within Launchpad are getting closed. If you would like us to re-investigate this issue, please create an issue in the php56u GitHub repository[1].

-Ben

[0] https://lists.launchpad.net/ius-community/msg01635.html
[1] https://github.com/iuscommunity-pkg/php56u