Metadata proxy server SSL handshake problem if Python >= 3

[Thomas Goirand]

Neutron API can now work over SSL using uwsgi or mod_wsgi since Rocky and beyond, though it looks like nothing has been thought for the proxy metadata server. Indeed, if using SSL, I'm getting the infamous SSL handshake problem:

X-Forwarded-For: 192.168.110.4
X-Neutron-Router-Id: 9b4c243d-51e4-45df-84b3-010b38a9e5bd __call__ /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:86
[-] Gotten ports for remote_address 192.168.110.4, network_id None, router_id 9b4c243d-51e4-45df-84b3-010b38a9e5bd are: [{'binding:host_id': 'swift01-compute-1.infomaniak.ch', 'description': '', 'device_id': '0acfcbbc-8365-4304-b9c1-c2b0161c9ff6', 'status': 'ACTIVE', 'network_id': '92783eba-7635-4639-aef9-5c8854c3d2f8', 'created_at': '2018-11-19T15:37:38Z', 'port_security_enabled': True, 'tenant_id': '1994b8ab99c1478ab0a3072ea0118474', 'device_owner': 'compute:nova', 'extra_dhcp_opts': [], 'name': '', 'binding:vif_type': 'ovs', 'binding:profile': {}, 'fixed_ips': [{'ip_address': '192.168.110.4', 'subnet_id': 'a573293e-d5b1-46dd-b301-80a29fc6b3c2'}], 'mac_address': 'fa:16:3e:02:b0:46', 'admin_state_up': True, 'revision_number': 5, 'binding:vnic_type': 'normal', 'binding:vif_details': {'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system'}, 'id': '2629ac29-a6d9-4526-9324-1798a7a7c785', 'tags': [], 'project_id': '1994b8ab99c1478ab0a3072ea0118474', 'security_groups': ['28e6a0d1-f8dc-48c4-a914-927bc07ecf88'], 'updated_at': '2018-11-19T15:37:44Z', 'allowed_address_pairs': [], 'qos_policy_id': None}] _get_instance_and_tenant_id /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:166

[-] Unexpected error.: ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 90, in __call__
   return self._proxy_request(instance_id, tenant_id, req)
 File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 198, in _proxy_request
   body=req.body)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1313, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1063, in _request
  (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 986, in _conn_request
   conn.connect()
 File "/usr/lib/python3.5/http/client.py", line 1261, in connect
   server_hostname=server_hostname)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 402, in wrap_socket
   return GreenSSLSocket(sock, *a, _context=self, **kw)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 91, in __init__
   self.do_handshake()
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 270, in do_handshake
   super(GreenSSLSocket, self).do_handshake)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
   return func(*a, **kw)
 File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
   self._sslobj.do_handshake()
 File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
   self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)

2018-11-20 08:43:52.966 2876 INFO eventlet.wsgi.server [-] 192.168.110.4,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 362 time: 0.2473052

Moreover, currently, we can only configure the metadata server to contact the API through a hostname and a port. This isn't sufficient at all. Indeed, in my case, the Nova API is running on:

https://api.example.com/compute

with haproxy rewriting the URL and removing "/compute" from the URL. Therefore, it's for me impossible to configure neutron-metadata correctly, unless setting nova-api to bind a non-SSL IP directly on the root, not protected by haproxy, which makes the whole thing completely insecure and not resilient to issue (ie: it's not HA anymore).

Please get all of this fixed ASAP, especially the Python 3 + Eventlet + SSL problem.

[YAMAMOTO Takashi (yamamoto)]

it seems the bug description contains at least two separate issues.
can you please consider to file separate bugs for separate issues?

[YAMAMOTO Takashi (yamamoto)]

i'm not familiar with "the infamous SSL handshake problem". can you elaborate a bit? and/or maybe provide a reference?

[Thomas Goirand (thomas-goirand)]

This is the issue I'm talking about:

https://github.com/eventlet/eventlet/issues/145

This has been known since 2014, and hasn't been resolved. This is due to monkeypatching from eventlet that isn't compatible with Python 3.