ifmap service is not up due to ssl version mismatch

[Nagendra Prasath]

In Rhel7.0 based contrail setups, during setup_all, API server is not up. API server didnt come up as IFmap service didnt come up.

ifmap-server.log:
2015-01-22 04:27:03,385 [main] DEBUG - ChannelAcceptor: Using keystore path= /etc/ifmap-server/keystore/irond.jks
2015-01-22 04:27:03,385 [main] DEBUG - ChannelAcceptor: Using trustStore path= /etc/ifmap-server/keystore/irond.jks
2015-01-22 04:27:03,796 [main] INFO - EventProcessor: Running with 4 workers and 2 forwarders
2015-01-22 04:27:03,797 [main] INFO - ActionProcessor: Running with 1 workers and 1 forwarders
2015-01-22 04:27:03,797 [main] INFO - ChannelAcceptor: Listening on port 8443 for incoming basic authentication connections
2015-01-22 04:27:03,799 [main] INFO - ChannelAcceptor: Listening on port 8444 for incoming certificate-based authentication connections
2015-01-22 04:27:03,809 [main] INFO - irond is running :-)
2015-01-22 04:27:03,816 [pool-5-thread-1] DEBUG - ChannelAcceptor: New connection from 10.1.1.2:48966 on port 8443
2015-01-22 04:27:03,869 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-01-22 04:27:03,869 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-01-22 04:27:03,869 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 10.1.1.2:48966:0 into state 'BROKEN'
2015-01-22 04:27:03,872 [pool-1-thread-1] DEBUG - EventProcessor: Processing BadChannelEvent on 10.1.1.2:48966:0
2015-01-22 04:27:03,872 [pool-1-thread-1] DEBUG - EventProcessor: No session found for 10.1.1.2:48966:0
2015-01-22 04:27:06,873 [pool-5-thread-1] DEBUG - ChannelAcceptor: New connection from 10.1.1.2:49113 on port 8443
2015-01-22 04:27:06,886 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-01-22 04:27:06,887 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)

contrail-api-0-stdout.log:
ERROR:cfgm_common.ifmap.client:Uknown error sending IF-MAP message to server
ERROR:cfgm_common.ifmap.client:Uknown error sending IF-MAP message to server
ERROR:cfgm_common.ifmap.client:Uknown error sending IF-MAP message to server
ERROR:cfgm_common.ifmap.client:Uknown error sending IF-MAP message to server

[Soren Hansen (soren)]

In case anyone is wondering what causes this all of a sudden: Recent security updates to openjdk removed SSLv3.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6845
Committed: http://github.org/Juniper/contrail-third-party/commit/c184e5d5efcb284dd3d3ff58e12a7285879980a2
Submitter: Zuul
Branch: R2.1

commit c184e5d5efcb284dd3d3ff58e12a7285879980a2
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:13:09 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ia54d4b4fbacb06717dee62a561bfbed8224a9b90
Closes-Bug: 1414790
(cherry picked from commit 244403e684ee42e857156e140e9ebbecd143b883)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6607
Committed: http://github.org/Juniper/contrail-third-party/commit/93779f843b11000644d3ff1cdf2dc9e6ea9ab88d
Submitter: Zuul
Branch: master

commit 93779f843b11000644d3ff1cdf2dc9e6ea9ab88d
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:13:09 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ia54d4b4fbacb06717dee62a561bfbed8224a9b90
Partial-Bug: 1414790

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6846
Committed: http://github.org/Juniper/contrail-controller/commit/2fbc58b3979683a73e4db1e87c4b406723c99a11
Submitter: Zuul
Branch: R2.1

commit 2fbc58b3979683a73e4db1e87c4b406723c99a11
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:15:15 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ieea5b157ce16b1526cd38e897da91af652add612
Closes-Bug: 1414790
(cherry picked from commit a968fe472257fe7d251010297e2bbae20cd48615)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6608
Committed: http://github.org/Juniper/contrail-controller/commit/5cd7b62344842c228906da0421f23dbb13d3c877
Submitter: Zuul
Branch: master

commit 5cd7b62344842c228906da0421f23dbb13d3c877
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:15:15 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ieea5b157ce16b1526cd38e897da91af652add612
Closes-Bug: 1414790
(cherry picked from commit a968fe472257fe7d251010297e2bbae20cd48615)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7187
Committed: http://github.org/Juniper/contrail-controller/commit/bcc5b226e1bba51a9f1afe39f321c4ad29e4f35a
Submitter: Zuul
Branch: R2.0

commit bcc5b226e1bba51a9f1afe39f321c4ad29e4f35a
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:15:15 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ieea5b157ce16b1526cd38e897da91af652add612
Closes-Bug: 1414790
(cherry picked from commit a968fe472257fe7d251010297e2bbae20cd48615)

[Maik Zumstrull (m-zumstrull)]

It looks like the fixes you have committed force the clients to use a strongly deprecated cipher suite (RC4-SHA), while being unusually permissive with regards to the protocol version (allowing anything down to SSLv2).

Have you considered enforcing TLS v1.0+ (or even v1.2+) as the protocol, and not forcing a particular cipher suite?

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7188
Committed: http://github.org/Juniper/contrail-third-party/commit/f2c526cbcdf656cee5249614b5bb0333ed4c3a31
Submitter: Zuul
Branch: R2.0

commit f2c526cbcdf656cee5249614b5bb0333ed4c3a31
Author: Sachin Bansal <email address hidden>
Date: Mon Jan 26 13:13:09 2015 -0800

Use SSLv23 for ifmap clients. SSLv3 is no longer supported.

Change-Id: Ia54d4b4fbacb06717dee62a561bfbed8224a9b90
Partial-Bug: 1414790

[Michael Renner (robe)]

To quote from draft-ietf-uta-tls-bcp-09 (https://tools.ietf.org/html/draft-ietf-uta-tls-bcp-09#page-9)

   o Implementations MUST NOT negotiate RC4 cipher suites.

      Rationale: The RC4 stream cipher has a variety of cryptographic
      weaknesses, as documented in [I-D.ietf-tls-prohibiting-rc4]. Note
      that DTLS specifically forbids the use of RC4 already.

[Michael Renner (robe)]

Furthermore:

   o Implementations MUST support TLS 1.2 [RFC5246] and MUST prefer to
      negotiate TLS version 1.2 over earlier versions of TLS.

      Rationale: Several stronger cipher suites are available only with
      TLS 1.2 (published in 2008). In fact, the cipher suites
      recommended by this document (Section 4.2 below) are only
      available in TLS 1.2.

   o Implementations MUST NOT negotiate SSL version 2.

      Rationale: Today, SSLv2 is considered insecure [RFC6176].

   o Implementations MUST NOT negotiate SSL version 3.

      Rationale: SSLv3 [RFC6101] was an improvement over SSLv2 and
      plugged some significant security holes, but did not support
      strong cipher suites. SSLv3 does not support TLS extensions, some
      of which (e.g., renegotiation_info) are security-critical. In
      addition, with the emergence of the POODLE attack [POODLE], SSLv3
      is now widely recognized as fundamentally insecure.

[Sachin Bansal (sbansal)]

Maik and Michael,

We understand your concerns about week cipher and ssl versions being enforced/accepted. We did try to use stronger versions, but because we support many different os flavors/versions, it didn't work for one or the other.

How about this as a solution: We provide ssl version and ciphers list as a configuration option and anyone who has concerns about security can set those values after making sure that they work with their installation?

Sachin Bansal

[Michael Renner (robe)]

Sachin,

thanks for your fast answer!

Exposing SSL/TLS configuration is an absolute must if your application supports or depends on it.

In my opinion the bare minimum would be configuration parameters for:

 * Certificate
 * Private Key
 * TLS protocols
 * TLS cipher string
 * verification of server certificates
 * verification of client certificates

And these should be implemented and documented for all contrail services using TLS for communication.

https://bettercrypto.org/static/applied-crypto-hardening.pdf offers a concise overview of TLS configuration parameters in other services if you need examples.

all the best,
Michael

[shweta (shweta-cse5)]

Hi,

We are using the latest code changes done for installing opencontrail using contrail-installer from this lnk https://github.com/juniper/contrail-installer,
 but still are are not able to start ifmap-server.
Please check below error logs for ifmap-server

Error logs while running ./contrail.sh start

2015-03-05 14:50:27 ++ screen -S contrail -p apiSrv -X stuff 'python /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py --conf_file /etc/contrail/contrail-api.conf --reset_config --rabbit_password contrail123 & echo $! >/home/contrail/shweta/contrail-installer/status/contrail/ap'Srv.pid; fg || echo "apiSrv failed to start" | tee "/home/contrail/shweta/contrail-installer/status/contrail/apiSrv.failure"
2015-03-05 14:50:27 ++ [[ apiSrv == \a\g\e\n\t ]]
2015-03-05 14:50:27 ++ echo 'Waiting for api-server to start...'
2015-03-05 14:50:27 ++ timeout 90 sh -c 'while ! http_proxy= wget -q -O- http://localhost:8082; do sleep 1; done'
2015-03-05 14:50:27 ++ echo 'api-server did not start'
2015-03-05 14:50:27 ++ exit 1
2015-03-05 14:50:27 ++ clean
2015-03-05 14:50:27 ++ local r=1
2015-03-05 14:50:27 ++ echo 'exited with status :1'
2015-03-05 14:50:27 ++ exit 1

Error logs of ifmap-server

2015-03-05 15:22:12,900 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:12,900 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03-05 15:22:12,900 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:43108:1232 into state 'BROKEN'
2015-03-05 15:22:15,905 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:15,905 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03-05 15:22:15,905 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:43109:1233 into state 'BROKEN'
2015-03-05 15:22:18,911 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:18,911 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03-05 15:22:18,911 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:43110:1234 into state 'BROKEN'
2015-03-05 15:22:21,912 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:21,913 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03-05 15:22:21,913 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:43112:1235 into state 'BROKEN'
2015-03-05 15:22:24,920 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:24,920 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03-05 15:22:24,921 [pool-6-thread-1] ERROR - ChannelThread: Setting channel 127.0.0.1:43114:1236 into state 'BROKEN'
2015-03-05 15:22:27,923 [pool-6-thread-1] ERROR - ChannelThread: Receiving request failed
2015-03-05 15:22:27,923 [pool-6-thread-1] ERROR - ChannelThread: SSLHandshakeException: Client doesn't know about our certificate (?)
2015-03...

[Stefan Andres (s-andres)]

The workaround for that:
Remove the following line:
/etc/java-7-openjdk/security/java.security:

jdk.tls.disabledAlgorithms=SSLv3

[shweta (shweta-cse5)]

Hi,

We removed that line and try to run it again but we are getting the same error.
One more thing while running the step ./contrail.sh build , we are getting java-6-openjdk instead of java-7-openjdk.
Is it ok to use java-6-openjdk instead of java-7-openjdk.

Thanks,

[Sachin Bansal (sbansal)]

Shweta, could you please also post content of contrail-api-0-stdout.log?

And yes, it should also work work with java-6-openjdk.

Sachin

[shweta (shweta-cse5)]

Hi Sachin,

Please find attached logs.

Thanks

[Sachin Bansal (sbansal)]

Hi Shweta,

You uploaded wrong file. I wanted contrail-api-0-stdout.log. And are you using ubuntu or centos/redhat?

Sachin

[shweta (shweta-cse5)]

Hi Sachin,

We are using Ubuntu to install opencontrail.

We are using below link to install opencontrail.
https://github.com/Juniper/contrail-installer

And we are stuck at ./contrail.sh start this step. contrail-api.log file is getting created here.
There is no file getting created with name contrail-api-0-stdout.log

Thanks

[Lamoni Finlayson (lamonif)]

Was there any resolution to this? I'm running into the same issues almost a year after the last post in this thread. I'm seeing the exact same error messages.

Thanks!