Neutron can't connect rabbitmq via ssl

[Long Quan Sha]

Recently, we update oslo_messaging from 1.4 to 1.6, then neutron and nova can't connect rabbitmq via ssl. I checked rabbitmq-server status, it works well. I can establish ssl connection by openssl. After some investigation, I decided to specify kombu_ssl_version in neutron.conf, then neutron server can connect rabbitmq by ssl, both of TLSv1, SSLv23 work for neutron.
It seems openstack components like nova and neutron has to specify kombu_ssl_version before ssl connection with rabbitmq,
but in oslo_messaging 1.4, it doesn't.

Do we have to explicitly specify kombu_ssl_version now ?

Neutron server error log:

2015-02-05 00:51:09.324 22999 ERROR oslo_messaging._drivers.impl_rabbit [req-cafa7fa7-488a-413f-90f4-6991f19a2c71 ] AMQP server 127.0.0.1:5671 closed the connection. Check login credentials: Socket closed

rabbitmq config:

# cat rabbitmq.config
[
  {kernel, [

  ]},
  {rabbit, [
    {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                    {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                    {verify,verify_none},
                    {fail_if_no_peer_cert,false}]},
    {tcp_listen_options, [binary, {packet,raw},
                                  {reuseaddr,true},
                                  {backlog,128},
                                  {nodelay,true},
                                  {exit_on_close,false},
                                  {keepalive,false}]},
    {default_user, <<"guest">>},
    {default_pass, <<"guest">>}
  ]}
].

[Long Quan Sha (shalq)]

In rabbitmq log, we found those ssl error :

=ERROR REPORT==== 5-Feb-2015::00:52:37 ===
error on AMQP connection <0.912.21>:
{ssl_upgrade_error,{tls_alert,"record overflow"}}

=ERROR REPORT==== 5-Feb-2015::00:52:37 ===
error on AMQP connection <0.916.21>:
{ssl_upgrade_error,{tls_alert,"record overflow"}}

=ERROR REPORT==== 5-Feb-2015::00:52:37 ===
error on AMQP connection <0.920.21>:
{ssl_upgrade_error,{tls_alert,"record overflow"}}

[Mehdi Abaakouk (sileht)]

https://review.openstack.org/#/c/154446/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154446
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=68cd8cfecc12a40b61a7b8557aad7b5a4e1f1c6d
Submitter: Jenkins
Branch: master

commit 68cd8cfecc12a40b61a7b8557aad7b5a4e1f1c6d
Author: Mehdi Abaakouk <email address hidden>
Date: Tue Feb 10 13:32:22 2015 +0100

    rabbit: Fix behavior of rabbit_use_ssl

    The regression around rabbit_use_ssl have been introduce when we
    have moved from the custom broker connection to the kombu one in:

     973301aa70527171749fa34897276c43898aeeb2

    This change fix the regression, it's now possible to use rabbit
    with ssl without having to set a version or a certificat again.

    Closes-bug: #1420164

    Change-Id: I8a2068cc433df0441e3c2b57c22c9b4558f8a29d

[Matt Riedemann (mriedem)]

Just to be clear, since I hit this also, if you apply the patch above on oslo.messaging 1.6.0 and set rabbit_use_ssl=True, you shouldn't set kombu_ssl_version unless you're setting the other kombu config options for like keyfile, certfile, and cafile. I still had kombu_ssl_version set and things were still broken even with the patch until I commented out kombu_ssl_version in my conf file.

[Davanum Srinivas (DIMS) (dims-v)]

Matt, is this the right behavior that you'd expect? Just to confirm :)