Synchronization fails when rabbitmq-server is configured with ssl=only

[Frode Nordahl]

ERROR * 08-27 07:50:10 [PID:25348] * root * Exception during kombu setup
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 404, in _setup_connection
    status_queue(self.conn.channel()).declare()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 266, in channel
    chan = self.transport.create_channel(self.connection)
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 282, in connect
    self.transport.connect()
  File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 109, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 150, in _connect
    self.sock.connect(sa)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

Although the error message is a bit confusing, the RabbitMQ server is indeed responding, but refusing non-SSL connections.

The RabbitMQ relation already provides the certificate, even if it would originate from vault:
root@juju-a08a15-zaza-2e151b8bbf8a-1:~# cat /etc/glance-simplestreams-sync/identity.yaml
[ snip ]

kombu_ssl_ca_certs: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNkekNDQWVDZ0F3SUJBZ0lVT0tTa2E5VEpySHU2TlFkMCtVMjZBZEFCeTNBd0RRWUpLb1pJaHZjTkFRRUYKQlFBd1J6RU5NQXNHQTFVRUNnd0VhblZxZFRFMk1EUUdBMVVFQ3d3dGNtRmlZbWwwYlhFdGMyVnlkbVZ5SUhObApjblpwWTJVZ1EyVnlkR2xtYVdOaGRHVWdRWFYwYUc5eWFYUjVNQjRYRFRFNU1EZ3lOekEyTXpJMU9Gb1hEVEkxCk1EZ3lOVEEyTXpJMU9Gb3dSekVOTUFzR0ExVUVDZ3dFYW5WcWRURTJNRFFHQTFVRUN3d3RjbUZpWW1sMGJYRXQKYzJWeWRtVnlJSE5sY25acFkyVWdRMlZ5ZEdsbWFXTmhkR1VnUVhWMGFHOXlhWFI1TUlHZk1BMEdDU3FHU0liMwpEUUVCQVFVQUE0R05BRENCaVFLQmdRRDFXK2haWVNEZlY4Uk9FQkIyNkJUcU5UTldxOEV0a1d1SnI3K25ST1JLCkdjSXo1cjBSKzRuODF2QWw2MHZFR0xTL1hnRkZnK2dyc0U5UTBkem9qaCtCeDJ3TjZCcmNiZnZhMVpaWHdlMUYKUFNUM1VrTzZHdUV2OWtxSmxvVjBYcFNmZVM4dWVvaTVERHJzcmRyTWVFbTBqTmpqTndWcElsZFFIY1J0U2dnVgphd0lEQVFBQm8yQXdYakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlNobHVwckpKQUFlbkg5Ck5tNlpuQ0M3L0dGVFdEQWZCZ05WSFNNRUdEQVdnQlNobHVwckpKQUFlbkg5Tm02Wm5DQzcvR0ZUV0RBTEJnTlYKSFE4RUJBTUNBUVl3RFFZSktvWklodmNOQVFFRkJRQURnWUVBUzdUTHNLWUg4L0poVFEvdlBHSmlpWitvTGthZAp1djNCcDN6SWF4YXZGUUlmU1FJWVZGUy9ZQThzbldHT1NPK0FpOUNXbzhJdWlYNjc2dk9NcWVWQkpuMEdKMTB5CjJMV0pyeHl3bzZFblZjL0J5bzlOV2lBU3lhV0xGa2luQU5VVmtuRXZQNnprb3QyM1VYRnBxU3dCVmd6VEp6bTcKKzI2UUF2VFRseDhMK1A0PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

I would suggest a quick fix for this is to just decode that cert and write it out to disk.

To have the kombu client use it we can change the code a long the lines of this:
diff --git a/scripts/glance-simplestreams-sync.py b/scripts/glance-simplestreams-sync.py
index 0a5d3ef..07567fa 100755
--- a/scripts/glance-simplestreams-sync.py
+++ b/scripts/glance-simplestreams-sync.py
@@ -396,7 +396,8 @@ class StatusExchange:
                                               host,
                                               id_conf['rabbit_virtual_host'])

- self.conn = kombu.BrokerConnection(url)
+ self.conn = kombu.BrokerConnection(
+ url, ssl={'ca_certs': '/tmp/kombu_ssl_ca_certs.pem'})
             self.exchange = kombu.Exchange("glance-simplestreams-sync-status")
             status_queue = kombu.Queue("glance-simplestreams-sync-status",
                                        exchange=self.exchange)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/678766

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678766
Committed: https://git.openstack.org/cgit/openstack/charm-glance-simplestreams-sync/commit/?id=24e426cd1609a32971b6efbaf0b941d52abf0aa1
Submitter: Zuul
Branch: master

commit 24e426cd1609a32971b6efbaf0b941d52abf0aa1
Author: Frode Nordahl <email address hidden>
Date: Tue Aug 27 11:16:33 2019 +0200

    Support connecting to RabbitMQ with TLS

    A side effect of the implementation is system installation of CA
    certificate from the RabbitMQ relation.

    This incidentally allows gss OpenStack clients to connect to API
    endpoints in a Vault TLS enabled deployment, as the Vault CA is
    announced over the RabbitMQ relation.

    Change-Id: I4317b5db112546c34b561f6c743e83504dda8694
    Closes-Bug: #1841553