Brocade driver HTTPS connection errors out with 'SSL3_WRITE_PENDING'

Bug #1653806 reported by Yucong Feng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Yucong Feng

Bug Description

When Brocade fabric setting 'fc_southbound_protocol' is set to 'HTTPS', any write attempts to the SAN switch will possibly encounter the error - ('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')

When communicating through a SSL connection, all data payloads must be in the string format. Currently the driver code is pulling in zoning data from the SAN switch through the method get_zone_info(). The zoning data retrieved can be unicode based, which when used in constructing the payload string to be sent during a write; a SSL3_WRITE_PENDING occurs since the string constructed for the payload is of type 'unicode'.

More info as to why a unicode string is not allowed, as it causes the buffer during the connection to be recreated each time, which is not allowed for the OpenSSL
https://github.com/shazow/urllib3/issues/717

2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/cinder/zonemanager/drivers/brocade/brcd_http_fc_zone_client.py", line 536, in add_zones
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager error_code, error_msg = self.post_zone_data(data)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/cinder/zonemanager/drivers/brocade/brcd_http_fc_zone_client.py", line 879, in post_zone_data
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager headers)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/cinder/zonemanager/drivers/brocade/brcd_http_fc_zone_client.py", line 110, in connect
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager verify=False)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in post
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager return self.request('POST', url, data=data, json=json, **kwargs)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager resp = self.send(prep, **send_kwargs)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 585, in send
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager r = adapter.send(request, **kwargs)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 403, in send
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager timeout=timeout
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager chunked=chunked)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 363, in _make_request
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager conn.request(method, url, **httplib_request_kw)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib64/python2.7/httplib.py", line 973, in request
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self._send_request(method, url, body, headers)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self.endheaders(body)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self._send_output(message_body)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib64/python2.7/httplib.py", line 833, in _send_output
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self.send(message_body)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib64/python2.7/httplib.py", line 805, in send
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self.sock.sendall(data)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 253, in sendall
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 242, in _send_until_done
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager return self.connection.send(data)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1271, in send
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager self._raise_ssl_error(self._ssl, result)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1187, in _raise_ssl_error
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager _raise_current_error()
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager raise exception_type(errors)
2016-12-27 15:19:43.972 31592 ERROR cinder.zonemanager.fc_zone_manager Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

Tags: fczm
Yucong Feng (yfeng)
Changed in cinder:
assignee: nobody → Yucong Feng (yfeng)
Changed in cinder:
status: New → In Progress
Revision history for this message
Yucong Feng (yfeng) wrote :
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/416371
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=777128dd34c125b01c359e31f129c6cc51be584e
Submitter: Jenkins
Branch: master

commit 777128dd34c125b01c359e31f129c6cc51be584e
Author: Yucong Feng <email address hidden>
Date: Tue Jan 3 23:44:38 2017 +0100

    Resolve Brocade HTTPS connection error

    Convert all payloads for SSL connections through https into string type.
    This prevents SSL3_WRITE_PENDING errors when communicating by SSL.

    Change-Id: I3d0e18e638593482e446a575da3dc52736e8e067
    Closes-Bug: #1653806

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.0.0rc1

This issue was fixed in the openstack/cinder 10.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.