cinder-volume and cinder-backup raise SSL error on "Notifying Schedulers of capabilities"

[Ryan Drew]

Hello! After using OSA to install cinder, I'm receiving a fatal error during the startup of both the cinder-volume and cinder-backup services in my cinder-volume lxc containers. The traceback is as follows:

---- BEGIN TRACEBACK ----

Jun 29 11:31:31 raisinbran-cinder-volumes-container-9b661c28 cinder-volume[9505]: 2020-06-29 11:31:31.470 9505 DEBUG cinder.manager [req-eb5d20f9-da33-4a28-addf-d6be955d52e9 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-package
s/cinder/manager.py:192
Jun 29 11:31:31 raisinbran-cinder-volumes-container-9b661c28 cinder-volume[9505]: 2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit [req-eb5d20f9-da33-4a28-addf-d6be955d52e9 - - - - -] Connection failed: [SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830) (retrying in 0 seconds):
ssl.SSLError: [SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/kombu/utils/functional.py", line 344, in retry_over_time
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/kombu/connection.py", line 283, in connect
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit return self.connection
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/kombu/connection.py", line 839, in connection
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection()
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/kombu/connection.py", line 794, in _establish_connection
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection()
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit conn.connect()
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/connection.py", line 311, in connect
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self.transport.connect()
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/transport.py", line 79, in connect
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self.socket_settings, self.read_timeout, self.write_timeout,
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/transport.py", line 189, in _init_socket
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self._setup_transport()
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/transport.py", line 304, in _setup_transport
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self.sock = self._wrap_socket(self.sock, **self.sslopts)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/transport.py", line 311, in _wrap_socket
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit return self._wrap_socket_sni(sock, **sslopts)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/amqp/transport.py", line 353, in _wrap_socket_sni
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit sock = ssl.wrap_socket(**opts)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/eventlet/green/ssl.py", line 414, in wrap_socket
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit return GreenSSLSocket(sock, *a, **kw)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/openstack/venvs/cinder-21.1.0.dev7/lib/python3.6/site-packages/eventlet/green/ssl.py", line 106, in __init__
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit ca_certs, do_handshake_on_connect and six.PY2, *args, **kw)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python3.6/ssl.py", line 701, in __init__
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self._context = SSLContext(ssl_version)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib64/python3.6/ssl.py", line 351, in __new__
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit self = _SSLContext.__new__(cls, protocol)
                                                                                  2020-06-29 11:31:31.497 9505 ERROR oslo.messaging._drivers.impl_rabbit ssl.SSLError: [SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)

---- END TRACEBACK ----

I've been looking everywhere I can on Google for something similar to this and am coming up short. I'm able to create the same same SSL context inside an interactive python session without fail.

Here's information about my setup:

* CentOS 7.8 Host running CentOS 7.8 LXC container.
* Python3.6 with virtualenv build by opentack ansible
* OpenSSL version 1.0.2k-fips (no SSL2 support)
* Kernel 3.10.0
* Relevant cinder.conf snippet:

[oslo_messaging_rabbit]
ssl = True
ssl_version = TLSv1_1

[oslo_messaging_notifications]
driver = messagingv2
transport_url = rabbit://cinder:[censored]@10.0.21.242:5671,cinder:[censored]@10.0.21.113:5671,cinder:[censored]@10.0.21.86:5671//cinder?ssl=1

* Installed packages in cinder venv:

Package Version
------------------------ -----------
alembic 1.4.2
amqp 2.5.2
appdirs 1.4.3
attrs 19.3.0
automaton 2.0.1
Babel 2.8.0
bcrypt 3.1.7
cachetools 4.1.0
castellan 3.0.1
certifi 2020.4.5.1
cffi 1.14.0
chardet 3.0.4
cinder 16.0.1.dev3
cliff 3.1.0
cmd2 0.8.9
cryptography 2.9
cursive 0.2.2
debtcollector 2.0.1
decorator 4.4.2
defusedxml 0.6.0
dnspython 1.15.0
dogpile.cache 0.9.0
ecdsa 0.15
eventlet 0.25.2
extras 1.0.0
fasteners 0.14.1
fixtures 3.0.0
futurist 2.1.1
google-api-core 1.16.0
google-api-python-client 1.8.0
google-auth 1.13.1
google-auth-httplib2 0.0.3
googleapis-common-protos 1.51.0
greenlet 0.4.15
httplib2 0.17.2
idna 2.9
importlib-metadata 1.6.0
iso8601 0.1.12
Jinja2 2.11.1
jmespath 0.9.5
jsonpatch 1.25
jsonpointer 2.0
jsonschema 3.2.0
keystoneauth1 4.0.0
keystonemiddleware 9.0.0
kombu 4.6.8
linecache2 1.0.0
lxml 4.5.0
Mako 1.1.2
MarkupSafe 1.1.1
monotonic 1.5
msgpack 0.6.2
munch 2.5.0
netaddr 0.7.19
netifaces 0.10.9
networkx 2.4
oauth2client 4.1.3
openstacksdk 0.46.0
os-brick 3.0.1
os-service-types 1.7.0
os-win 5.0.1
osc-lib 2.0.0
oslo.cache 2.3.0
oslo.concurrency 4.0.2
oslo.config 8.0.2
oslo.context 3.0.2
oslo.db 8.1.0
oslo.i18n 4.0.1
oslo.log 4.1.1
oslo.messaging 12.1.0
oslo.middleware 4.0.2
oslo.policy 3.1.0
oslo.privsep 2.1.1
oslo.reports 2.0.1
oslo.rootwrap 6.0.2
oslo.serialization 3.1.1
oslo.service 2.1.1
oslo.upgradecheck 1.0.1
oslo.utils 4.1.1
oslo.versionedobjects 2.0.2
oslo.vmware 3.3.1
osprofiler 3.1.0
paramiko 2.7.1
Paste 3.4.0
PasteDeploy 2.1.0
pbr 5.4.5
pip 20.1.1
prettytable 0.7.2
protobuf 3.11.3
psutil 5.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycadf 3.0.0
pycparser 2.20
pydot 1.4.1
pyinotify 0.9.6
pymemcache 3.1.0
PyMySQL 0.9.3
PyNaCl 1.3.0
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyperclip 1.8.0
pyrsistent 0.16.0
python-barbicanclient 4.10.0
python-cinderclient 7.0.0
python-dateutil 2.8.1
python-editor 1.0.4
python-glanceclient 3.1.1
python-keystoneclient 4.0.0
python-memcached 1.59
python-mimeparse 1.6.0
python-novaclient 17.0.0
python-openstackclient 5.2.0
python-swiftclient 3.9.0
pytz 2019.3
pyudev 0.22.0
PyYAML 5.3.1
repoze.lru 0.7
requests 2.23.0
requestsexceptions 1.4.0
retrying 1.3.3
rfc3986 1.4.0
Routes 2.4.1
rsa 4.0
rtslib-fb 2.1.71
setuptools 46.4.0
simplejson 3.17.0
six 1.14.0
SQLAlchemy 1.3.16
sqlalchemy-migrate 0.13.0
sqlparse 0.3.1
statsd 3.3.0
stevedore 1.32.0
suds-jurko 0.6
systemd-python 234
tabulate 0.8.7
taskflow 4.1.0
Tempita 0.5.2
tenacity 6.1.0
testresources 2.0.1
testscenarios 0.5.0
testtools 2.4.0
tooz 2.3.0
traceback2 1.4.0
unittest2 1.1.0
uritemplate 3.0.1
urllib3 1.25.8
vine 1.3.0
voluptuous 0.11.7
warlock 1.3.3
wcwidth 0.1.9
WebOb 1.8.6
wheel 0.34.2
wrapt 1.12.1
yappi 1.2.3
zipp 3.1.0

In order to get cinder-volume to startup, I've had to just disable SSL. This is super confusing as, why would I get an SSL2 related error when SSL2 isn't even being used?

I'm not sure if this is in error in the cinder package itself or somewhere down the chain the rabbit client implementation, but I thought I'd raise this bug here as a start.

Thanks!