VNX: driver cannot start under py37 due to wrap_socket() got an unexpected keyword argument '_context'

Bug #1833160 reported by Ryan Liang on 2019-06-18
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Low
Unassigned

Bug Description

VNX driver is using library urllib to do http connection to the storage system.

But under python 37, it failed to start due to some issues like wrap_socket() got an unexpected keyword argument '_context'. This issue is reported in the eventlet repo and still open: https://github.com/eventlet/eventlet/issues/526

Versions:
Python 3.7.3
six 1.12.0
urllib3 11.24.3
eventlet 0.25.0

VNX manila driver log:
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager [None req-d2b44d43-5ba8-489b-a002-d4350d68bdc8 None None] Error encountered during initialization of driver EMCShareDriver@vnxmanila2@london: TypeError: wrap_socket() got an unexpected keyword argument '_context'
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager Traceback (most recent call last):
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/manager.py", line 310, in _driver_setup
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.driver.do_setup(ctxt)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/driver.py", line 155, in do_setup
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.plugin.connect(self, context)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/common/enas/utils.py", line 52, in inner
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager ret = func(self, *args, **kwargs)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/plugins/vnx/connection.py", line 584, in connect
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.manager = manager.StorageObjectManager(config)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/plugins/vnx/object_manager.py", line 44, in __init__
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.connectors['XML'] = connector.XMLAPIConnector(configuration)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/common/enas/connector.py", line 55, in __init__
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self._do_setup()
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/dell_emc/common/enas/connector.py", line 63, in _do_setup
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager resp = self.url_opener.open(req)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 525, in open
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager response = self._open(req, data)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager '_open', req)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager result = func(*args)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 1360, in https_open
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager context=self._context, check_hostname=self._check_hostname)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 1317, in do_open
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager encode_chunked=req.has_header('Transfer-encoding'))
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 1229, in request
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self._send_request(method, url, body, headers, encode_chunked)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.endheaders(body, encode_chunked=encode_chunked)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self._send_output(message_body, encode_chunked=encode_chunked)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 1016, in _send_output
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.send(msg)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 956, in send
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager self.connect()
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/lib/python3.7/http/client.py", line 1392, in connect
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager server_hostname=server_hostname)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/local/lib/python3.7/dist-packages/eventlet/green/ssl.py", line 438, in wrap_socket
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager return GreenSSLSocket(sock, *a, _context=self, **kw)
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager File "/usr/local/lib/python3.7/dist-packages/eventlet/green/ssl.py", line 76, in __new__
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager *args, **kw
Jun 17 09:47:20 vnxmanila2 manila-share[15024]: ERROR manila.share.manager TypeError: wrap_socket() got an unexpected keyword argument '_context'

Tom Barron (tpb) on 2019-06-18
tags: added: dell-emc driver python3 vnx
Jason Grosso (jgrosso) wrote :

Hey Ryan is there dell-emc engineer we can assign this driver issues too? or are you taking it?

Ryan Liang (ryan-liang) wrote :

Actually, it isn't an issue of dellemc driver. It looks more like the eventlet issue: https://github.com/eventlet/eventlet/issues/526.

The reason why I filed this issue is to track the same issue which could meet by other users/in other drivers.

BTW, do you notice if there is any other driver meeting the same issue? What are their fixings or workarounds?

Tom Barron (tpb) wrote :
Download full text (3.9 KiB)

Hmm, I see urllib used in the following:

tbarron@localhost manila (master)$ grep -r urllib manila/share/drivers
manila/share/drivers/quobyte/jsonrpc.py:import six.moves.urllib.parse as urlparse
manila/share/drivers/huawei/v3/helper.py: requests.packages.urllib3.disable_warnings(
manila/share/drivers/huawei/v3/helper.py: requests.packages.urllib3.exceptions.InsecureRequestWarning)
manila/share/drivers/huawei/v3/helper.py: requests.packages.urllib3.disable_warnings(
manila/share/drivers/huawei/v3/helper.py: requests.packages.urllib3.exceptions.InsecurePlatformWarning)
manila/share/drivers/nexenta/ns5/jsonrpc.py:from requests.packages.urllib3 import exceptions
manila/share/drivers/nexenta/ns5/jsonrpc.py:requests.packages.urllib3.disable_warnings(exceptions.InsecureRequestWarning)
manila/share/drivers/nexenta/ns5/jsonrpc.py:requests.packages.urllib3.disable_warnings(
manila/share/drivers/qnap/api.py:from six.moves import urllib
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params, True)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/qnap/api.py: sanitized_params = urllib.parse.urlencode(sanitized_params)
manila/share/drivers/zfssa/restclient.py:from six.moves.urllib import error as urlerror
manila/share/drivers/zfssa/restclient.py:from six.moves.urllib import request as urlrequest
manila/share/drivers/zfssa/restclient.py: """ZFSSA urllib client."""
manila/share/drivers/netapp/dataontap/client/api.py:from six.moves import urllib
manila/share/drivers/netapp/dataontap/client/api.py: except urllib.error.HTTPError as e:
manila/share/drivers/netapp/dataontap/client/api.py: except urllib...

Read more...

Tom Barron (tpb) wrote :

Netapp folks report that they aren't yet running CI with python 3.7 but will check to see if they have the same issue under 3.7.

Carlos Eduardo (silvacarlose) wrote :

Hey Ryan, we also use the library urllib to communicate with the storage in the NetApp driver. I've created an environment that uses python 3.7 and tested the communication with our storage. Everything seems working fine.

Carlos Eduardo (silvacarlose) wrote :

Here are some command outputs:
pythirty-seven-test:/opt/stack/new/manila$ python3 --version
Python 3.7.3

Here is an example of a returned value from the storage:
Jul 02 12:40:39.549043 pythirty-seven-test manila-share[12663]: DEBUG manila.share.drivers.netapp.dataontap.client.api [None req-8eaf5be4-1a24-44f0-9ecb-31a381f41a4f None None] Request: b'<netapp xmlns="http://www.netapp.com/filer/admin" version="1.140">\n <aggr-get-iter>\n <query>\n <aggr-attributes>\n <aggregate-name>aggr1</aggregate-name>\n </aggr-attributes>\n </query>\n <desired-attributes>\n <aggr-attributes>\n <aggregate-name/>\n <aggr-ownership-attributes>\n <home-name/>\n </aggr-ownership-attributes>\n </aggr-attributes>\n </desired-attributes>\n <max-records>50</max-records>\n </aggr-get-iter>\n</netapp>\n' {{(pid=12685) invoke_elem /opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py:242}}

Ryan Liang (ryan-liang) wrote :

What version of eventlet and urllib are you using?

Ryan Liang (ryan-liang) wrote :

This issue occurs when below conditions meet together:
1. Python version is 3.7.3.
2. Using python/urllib (via six.moves.urllib or other ways).
3. Using https connection where ssl is involved.
4. Patching via eventlet.

The reason is that ssl.wrap_socket function prototype changes from python3.7.3, `_context` is not a parameter anymore. Check source codes /usr/local/lib/python3.7/ssl.py for detail.
However, eventlet doesn't change the prototype accordingly.

So, you'll see the error `TypeError: wrap_socket() got an unexpected keyword argument '_context'` above when eventlet does the patch.

Carlos Eduardo (silvacarlose) wrote :

Hi, Ryan! I'm so sorry I've missed the question you asked. Here are the versions of urllib, eventlet and six I used in the environment to test the NetApp driver against py37.

urllib:
Name: urllib3
Version: 1.25.3
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: <email address hidden>
License: MIT
Location: /usr/local/lib/python3.7/dist-packages

eventlet:
Name: eventlet
Version: 0.25.0
Summary: Highly concurrent networking library
Home-page: http://eventlet.net
Author: Linden Lab
Author-email: <email address hidden>
License: UNKNOWN
Location: /usr/local/lib/python3.7/dist-packages
Requires: greenlet, six, dnspython, monotonic

six:
Name: six
Version: 1.12.0
Summary: Python 2 and 3 compatibility utilities
Home-page: https://github.com/benjaminp/six
Author: Benjamin Peterson
Author-email: <email address hidden>
License: MIT
Location: /usr/local/lib/python3.7/dist-packages
Requires:

These outputs I received while running the command tox show `package_name`
I've realized that our versions (urllib and eventlet) are very different. I tried to check for upgrades for each package I've pasted here but pip says I'm already up to date.
Let me know if I can help you with something.
And sorry for the delay again

Carlos Eduardo (silvacarlose) wrote :

s/while running the command tox show/while running the command pip show

Ryan Liang (ryan-liang) wrote :

Hi Carlos, which version of python are you using? Could you give the full version number? python3.7.2 or python3.7.3 or others?

Ryan Liang (ryan-liang) wrote :
Download full text (3.4 KiB)

Oh, I see the version of python you are using. It's python3.7.3. Forget my last comment.
Are you using https connection? If yes, it doesn't make sense.

In the source of eventlet (0.25.0), /usr/local/lib/python3.7/site-packages/eventlet/green/ssl.py:

431 if hasattr(__ssl, 'SSLContext'):
432 _original_sslcontext = __ssl.SSLContext
433
434 class GreenSSLContext(_original_sslcontext):
435 __slots__ = ()
436
437 def wrap_socket(self, sock, *a, **kw):
438 return GreenSSLSocket(sock, *a, _context=self, **kw) # <<<<< _context passed in here

 43 class GreenSSLSocket(_original_sslsocket):
 57 def __new__(cls, sock=None, keyfile=None, certfile=None,
 58 server_side=False, cert_reqs=CERT_NONE,
 59 ssl_version=PROTOCOL_SSLv23, ca_certs=None,
 60 do_handshake_on_connect=True, *args, **kw): # <<<<< _context passed in via **kw
 61 if _is_under_py_3_7:
 62 return super(GreenSSLSocket, cls).__new__(cls)
 63 else:
 64 if not isinstance(sock, GreenSocket):
 65 sock = GreenSocket(sock)
 66 with _original_ssl_context():
 67 ret = _original_wrap_socket(
 68 sock=sock.fd,
 69 keyfile=keyfile,
 70 certfile=certfile,
 71 server_side=server_side,
 72 cert_reqs=cert_reqs,
 73 ssl_version=ssl_version,
 74 ca_certs=ca_certs,
 75 do_handshake_on_connect=False,
 76 *args, **kw # <<<<< _context passed in via **kw
 77 )

  1 __ssl = __import__('ssl')
 25 _original_wrap_socket = __ssl.wrap_socket

But in python37's /usr/local/lib/python3.7/ssl.py

1215 def wrap_socket(sock, keyfile=None, certfile=None, # <<<<< no _context
1216 server_side=False, cert_reqs=CERT_NONE,
1217 ssl_version=PROTOCOL_TLS, ca_certs=None,
1218 do_handshake_on_connect=True,
1219 suppress_ragged_eofs=True,
1220 ciphers=None):

I made a quick workaround in eventlet, /usr/local/lib/python3.7/site-packages/eventlet/green/ssl.py.
Then it works.

 57 def __new__(cls, sock=None, keyfile=None, certfile=None,
 58 server_side=False, cert_reqs=CERT_NONE,
 59 ssl_version=PROTOCOL_SSLv23, ca_certs=None,
 60 do_handshake_on_connect=True, *args, **kw):
 61 if _is_under_py_3_7:
 62 return super(GreenSSLSocket, cls).__new__(cls)
 63 else:
 64 if not isinstance(sock, GreenSocket):
 65 sock = GreenSocket(sock)
 66 with _original_ssl_context():
 67 for ignore in ('_context', 'server_hostname'): # <<<<< remove _context and server_hostname
 68 if ignore in kw:
 69 kw.pop(ignore)
 70 ret = _original_wrap_socket(
 71 sock=sock.fd,
 72 keyfile=keyfile,
 73 certfile=certfile,
 7...

Read more...

Carlos Eduardo (silvacarlose) wrote :

I tested with both HTTP and HTTPS. The first test I've performed was with HTTP, it worked perfectly. Then I've manually changed the driver to work with HTTPS and the communication is also working fine.

Ryan Liang (ryan-liang) wrote :

I wrote a sample code with `eventlet`, `urllib` and `python37`: https://gist.github.com/Murray-LIANG/9b506d98cf76d633a9d0e82f2bb5ece5

It raises the same exception as reported by this bug:
    TypeError: wrap_socket() got an unexpected keyword argument '_context'

And the sample code is like the one in netapp dataontap driver (manila/share/drivers/netapp/dataontap/client/api.py).
Both of them use `urllib.request.HTTPBasicAuthHandler` to open requests.

I have no netapp device, so I can't figure out how netapp dataontap driver works around this issue.

The environment I am using:
Python 3.7.2, 3.7.3 and 3.7.4
eventlet==0.25.0
six==1.12.0

Carlos Eduardo (silvacarlose) wrote :

Hey, Ryan! I've tested this script you created in my environment and I had the same problem you're facing. The stacktrace I got while running the script is:

... resp = opener.open(req)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python3.7/dist-packages/eventlet/green/ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/usr/local/lib/python3.7/dist-packages/eventlet/green/ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'
>>> print(resp)

I've looked how the NetApp driver works it around since I'm using the same library versions you're using and the driver is not breaking. I didn't found things that are strongly different.
I've added a log to show as a dict the created request object. Here is my output:

{'_full_url': 'http://x.x.x.x:80/servlets/netapp.servlets.admin.XMLrequest_filer', 'fragment': None, 'type': 'http', 'host': 'x.x.x.x:80', 'selector': '/servlets/netapp.servlets.admin.XMLrequest_filer', 'headers': {'Content-type': 'text/xml', 'Charset': 'utf-8'}, 'unredirected_hdrs': {}, '_data': b'<netapp xmlns="http://www.netapp.com/filer/admin" version="1.140"><perf-object-counter-list-info><objectname>processor</objectname></perf-object-counter-list-info></netapp>', '_tunnel_host': None, 'origin_req_host': 'x.x.x.x', 'unverifiable': False}

You can see if there is something different.
Also, does the VNX driver mounts the request url with a port?
You can take a look in the way NetApp driver does it [1]. There are few different things I've realized while comparing the way each driver create the requests. I think you're very close to figure out a solution to this problem.
[1] https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/client/api.py#L230

Ryan Liang (ryan-liang) wrote :

So you're still using http instead of https to connect to netapp device.
'_full_url': 'http://x.x.x.x:80/servlets/netapp.servlets.admin.XMLrequest_filer'
'type': 'http'

Could you switch it to https and have another try?
urllib will use different ways to open the connection when the `type` is diffferent (http or https).

Carlos Eduardo (silvacarlose) wrote :
Download full text (10.6 KiB)

I haven't noticed that the driver was not using https to make its calls. Thanks for pointing it out. There are two points where we set http or https. Now I've forced it in both places and I had the same mistake you're having. If I use HTTP instead of HTTPS, it works correctly.

Aug 22 17:57:44.809258 37-carlos-pythirty-seven-test manila-share[1481]: WARNING manila.share.drivers.netapp.dataontap.client.api [None req-3970f319-d985-46ac-9d03-50211f861978 None None] {'_full_url': 'https://x.x.x.x:80/servlets/netapp.servlets.admin.XMLrequest_filer', 'fragment': None, 'type': 'https', 'host': 'x.x.x.x:80', 'selector': '/servlets/netapp.servlets.admin.XMLrequest_filer', 'headers': {'Content-type': 'text/xml', 'Charset': 'utf-8'}, 'unredirected_hdrs': {}, '_data': b'<netapp xmlns="https://www.netapp.com/filer/admin" version="1.15"><system-get-ontapi-version/></netapp>', '_tunnel_host': None, 'origin_req_host': 'x.x.x.x', 'unverifiable': False}
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager [None req-3970f319-d985-46ac-9d03-50211f861978 None None] Error encountered during initialization of driver NetAppCmodeMultiSvmShareDriver@37-carlos-pythirty-seven-test@netapp2: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - unknown:wrap_socket() got an unexpected keyword argument '_context'
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager Traceback (most recent call last):
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py", line 254, in invoke_elem
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager response = self._opener.open(request)
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 525, in open
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager response = self._open(req, data)
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager '_open', req)
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager result = func(*args)
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/usr/lib/python3.7/urllib/request.py", line 1360, in https_open
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager context=self._context, check_hostname=self._check_hostname)
Aug 22 17:57:44.815719 37-carlos-pythirty-seven-test manila-share[1481]: ERROR manila.share.manager File "/u...

Tom Barron (tpb) wrote :

Do the ONTAP and VNX cinder drivers have the same issue?

Ryan Liang (ryan-liang) wrote :

VNX cinder driver doesn't have this issue. Because VNX cinder uses naviseccli via ssh to do VNX block related operations. And VNX manila uses XML api based on http/https.

It seems this issue happens when using urllib and eventlet. Our Unity Cinder and Manila drivers are using requests library and haven't met this issue.

Carlos Eduardo (silvacarlose) wrote :

Hey, Tom! As the NetApp Manila driver, NetApp Cinder driver also uses urllib to do its calls. I haven't tested it yet but I'm almost 100% sure that the same will happen with the NetApp Cinder driver.

Carlos Eduardo (silvacarlose) wrote :

At that time (when we were discussing) we were able to identify the issue in our side. We have started looking for some possible fixes. Felipe Rodrigues will update this bug with his updates soon.

Felipe Rodrigues (felipefutty) wrote :

I could reproduce the bug with netapp driver!

As it is state in the last line: the ssl.py patched by the eventlet is not expecting an argument called "_context" to wrap_socket(). This error happens because ssl library has changed from python 3.6 to python 3.7 as showed by [1], but the eventlet patching has not.

The eventlet bug with python3.7 as well-known by the issue 526 [2]. This issue was open more than one year ago. There is a long discussion about it! Summing up it: there is an attempt to fix it by a pull request [3], it was created may 2019 and it has not finished yet, though! Actually, since November 2019, no more updates on that. So, the eventlet bug will not be fixed early!
However, in the eventlet disscussion, it was pointed 2 possible solutions:

1. Replace the eventlet by gevent

2. Replace “ssl” by “pyopenssl” lib

The code [6] reproduces the bug in a small peace of code. It uses the same library and methods as netapp driver does. It can be used to test the solutions.

Just replacing the eventlet by gevent and running the code [4], the bug is solved, validating the solution “1”! For the manila scenario, it might not be feasible, because the patching is applied out of driver scope, affecting all vendors. So, this solution is skipped by now (maybe a discussion about eventlet vs gevent could be created).

The solution “2” is not applied for the code [6], because it uses urllib (from six.moves). The urllib is a library that comes with python and its secure connection is mandatory from ssl library. I haven’t found a way to replace the ssl by pyopenssl in urllib. Hence, for the manila driver we could not apply the “2”.

The possible solution would be change the request library: instead of using urllib, it uses a lib that can perform ssl from pyopenssl. The “requests” library does it! For this one, the default is “pyopenssl” if this library is installed, otherwise it would use the “ssl” lib.
To validate the “requests” solution, the following code is created:

import eventlet
requests = eventlet.import_patched(“requests”)
full_url = “https://www.wikipedia.org
resp = requests.get(full_url)
print(rest)

Without installing the “pyopenssl”, running the above code with python3.7 the same reported bug happens. After installing the “pyopenssl” (with pip3.7), the bug is fixed. The reason is that the “requests” lib uses the patched “pyopenssl” instead of the broken patched “ssl”.

Summing up, the possible solution would be change the request library: instead of std python lib “urllib”, uses the third party lib “requests”!

[1] https://bugs.python.org/issue32951
[2] https://github.com/eventlet/eventlet/issues/526
[3] https://github.com/eventlet/eventlet/pull/565
[4] https://gist.github.com/Murray-LIANG/9b506d98cf76d633a9d0e82f2bb5ece5

Ryan Liang (ryan-liang) wrote :

The new version of requests 2.24.0 replaced `pyopenss` with `ssl`.
So, any vendor driver (no matter Cinder or Manila) which are using requests 2.24.0 will have the same issue.

Felipe Rodrigues (felipefutty) wrote :

As the discussion of this new feature shows [1]: it can forces the use of `pyopenssl` by calling:

urllib3.contrib.pyopenssl.inject_into_urllib3()

[1] https://github.com/psf/requests/pull/5443#issuecomment-645740394

Vida Haririan (vhariria) on 2020-08-20
Changed in manila:
importance: Undecided → Low
Felipe Rodrigues (felipefutty) wrote :

As the research showed, the source of the bug is the bug in the eventlet library, while patching the ssl library in py3.7 (or newer). The eventlet has released a new version (0.26) that fixes that bug, though!

The link of the new release: https://github.com/eventlet/eventlet/releases/tag/v0.26.0
It mentions: "Fix compatibility with SSLContext usage >= Python 3.7"

I tested the 0.26 eventlet with py3.7 for requesting a https content and it worked!

Sam Wan (sam-wan) wrote :

Verified the issue has been fixed.
now powermax/vnx manila CI's can run under python3.7

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.