requests to SSL wrapped sockets hang while reading using py3

Bug #1482633 reported by Valeriy Ponomaryov on 2015-08-07
24
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance
Undecided
Unassigned
Manila
Low
Victoria Martinez de la Cruz
OpenStack Compute (nova)
Undecided
Unassigned
neutron
High
Unassigned
oslo.service
High
Unassigned

Bug Description

If we run unit tests using py3 then we get following errors:

======================================================================
FAIL: manila.tests.test_wsgi.TestWSGIServer.test_app_using_ssl
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  pythonlogging:''
  stdout

stderr: {{{
Traceback (most recent call last):
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/wsgi.py", line 823, in server
    client_socket = sock.accept()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 333, in accept
    suppress_ragged_eofs=self.suppress_ragged_eofs)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 88, in __init__
    self.do_handshake()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 241, in do_handshake
    super(GreenSSLSocket, self).do_handshake)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 106, in _call_trampolining
    return func(*a, **kw)
  File "/usr/lib/python3.4/ssl.py", line 805, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:598)
}}}

Traceback (most recent call last):
  File "/home/vponomaryov/Documents/python/projects/manila/manila/tests/test_wsgi.py", line 181, in test_app_using_ssl
    'https://127.0.0.1:%d/' % server.port)
  File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 455, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 473, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1273, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.4/urllib/request.py", line 1232, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1065, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1103, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1061, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 906, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 841, in send
    self.connect()
  File "/usr/lib/python3.4/http/client.py", line 1205, in connect
    server_hostname=server_hostname)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 362, in _green_sslcontext_wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 88, in __init__
    self.do_handshake()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 241, in do_handshake
    super(GreenSSLSocket, self).do_handshake)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/green/ssl.py", line 116, in _call_trampolining
    timeout_exc=timeout_exc('timed out'))
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
    return hub.switch()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/hub.py", line 346, in run
    self.wait(sleep_time)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/poll.py", line 85, in wait
    presult = self.do_poll(seconds)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
    return self.poll.poll(seconds)
  File "/home/vponomaryov/Documents/python/projects/manila/.tox/py34/lib/python3.4/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException

Debugging by hands shown that service starting and closing OK, but when we request something from it using py3 and SSL then we do not get answer at all. In case of unit tests we face test timeout.

Reviewed: https://review.openstack.org/210372
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=596b3405629f04f7079106cb5ae3e8324a4361bc
Submitter: Jenkins
Branch: master

commit 596b3405629f04f7079106cb5ae3e8324a4361bc
Author: Valeriy Ponomaryov <email address hidden>
Date: Fri Aug 7 15:41:47 2015 +0300

    Skip unit tests for SSL + py3

    When we request something from SSL wrapped app using py3 and we do not get
    answer at all. In case of unit tests we face test timeout.
    So, skip these tests until related b_u_g is fixed.

    Change-Id: I809668e5c04b7ace45291e3d1bfd743853f55918
    Related-Bug: #1482633
    Partially-Implements: bp py3-compatibility

Changed in manila:
importance: Undecided → Low

Neutron also hangs in py34 when running neutron.tests.unit.test_wsgi.TestWSGIServerWithSSL.

Changed in neutron:
status: New → Confirmed
importance: Undecided → High

Reviewed: https://review.openstack.org/219389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ef409d9da2ecbf12f9916a9a933231146538cf04
Submitter: Jenkins
Branch: master

commit ef409d9da2ecbf12f9916a9a933231146538cf04
Author: Ihar Hrachyshka <email address hidden>
Date: Tue Sep 1 21:45:55 2015 +0200

    Enable most unit tests for py34 job

    * Skip TestWSGIServerWithSSL[1] for Python 3 since it seems wsgi + ssl +
      eventlet setup does not behave correctly now,
    * Skip test_json_with_utf8[2] until we solve unicode/utf8 encode/decode,
    * Fix some more tests to pass for py3,
    * Replace print by print() in docs/docstrings.

    [1] neutron.tests.unit.test_wsgi (bug 1482633)
    [2] neutron.tests.unit.test_wsgi.JSONDictSerializerTest (bug 1491824)

    Related-Bug: #1482633
    Related-Bug: #1491824
    Blueprint: neutron-python3
    Co-Authored-By: Cyril Roelandt <email address hidden>
    Co-Authored-By: Cedric Brandily <email address hidden>
    Co-Authored-By: sonu.kumar <email address hidden>
    Change-Id: I26e513d4dcf473f4cd79728382fc94af3d901b5d

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Elena Ezhova (eezhova) wrote :

Neutron is going to consume ssl functionality fron oslo.service, so this bug is no longer valid for neutron.

Changed in neutron:
status: Confirmed → Invalid

@Elena: is there a blueprint/patch series about this upcoming change?

Elena Ezhova (eezhova) wrote :

@Cyril, Yes, here is a change: https://review.openstack.org/#/c/227835/ . It has the Related-Bug tag, so I thought Launchpad would link it automatically.

Reviewed: https://review.openstack.org/227835
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=216d2d0b7594ad05916bcee6ed8f80f1e0b385d0
Submitter: Jenkins
Branch: master

commit 216d2d0b7594ad05916bcee6ed8f80f1e0b385d0
Author: Elena Ezhova <email address hidden>
Date: Fri Sep 25 16:07:51 2015 +0300

    Consume sslutils and wsgi modules from oslo.service

    sslutils and basic WSGI functionality have been moved to
    oslo.service and now Neutron can reuse them.

    Marked ssl options that were renamed in oslo.service as
    deprecated.

    Added a note about possible implications for out-of-tree plugins
    to neutron_api.rst

    Bumped oslo.service version to 0.9.0.

    Related-Bug: #1482633

    Depends-On: I0424a6c261fae447dbc25b3abf00258c860a88f5
    Change-Id: Ibfdf07e665fcfcd093a0e31274e1a6116706aec2

Sean Dague (sdague) wrote :

Is there an upstream eventlet bug for this?

Changed in nova:
status: New → Invalid
Sean Dague (sdague) wrote :

Also on oslo.service now, so not actually a nova bug any more

Changed in manila:
milestone: none → newton-1
status: New → Triaged
Elena Ezhova (eezhova) wrote :

The related oslo.service tests no longer hang with python 3.4 but fail with http://paste.openstack.org/show/529631/

Reviewed: https://review.openstack.org/426505
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1b7191f4f2cf32254ba4b62dca219de330ef2e43
Submitter: Jenkins
Branch: master

commit 1b7191f4f2cf32254ba4b62dca219de330ef2e43
Author: ChangBo Guo(gcb) <email address hidden>
Date: Sat Jan 28 21:33:45 2017 +0800

    Skip unit tests for SSL + py3

    When we request something from SSL wrapped app using py3 and we do
    not get answer at all. In case of unit tests we face test timeout.
    So, skip these tests until related bug is fixed. This also enables
    other test cases in this module.

    Related-Bug: #1482633
    Partially-Implements: blueprint goal-python35

    Change-Id: I4933e47b0759716f387ec26e3bd8a4e725eb4f91

Corey Bryant (corey.bryant) wrote :

I appear to be hitting this same issue with novapackage builds on Ubuntu Artful with py2.

http://paste.ubuntu.com/25378049/

Changed in manila:
assignee: nobody → Victoria Martinez de la Cruz (vkmc)

I couldn't reproduce this with python3.5. I think it is safe to make python3 gate voting again.

Ben Nemec (bnemec) wrote :

Is there anything left to do on this from the oslo.service side? I think the python 3 unit tests are running almost everywhere now so I'm inclined to believe it was fixed.

Changed in oslo.service:
status: New → Incomplete
Ben Nemec (bnemec) wrote :

Oh, my mistake. I see the oslo.service tests are being skipped on py3. We need to stop doing that and figure out how to fix this soon. Without it we don't have proper python 3 support in oslo.service.

Changed in oslo.service:
status: Incomplete → Triaged
importance: Undecided → High
Zane Bitter (zaneb) wrote :

This appears to be a series of long-standing bugs in eventlet:

Python 3.5 failure mode:
https://github.com/eventlet/eventlet/issues/308
https://github.com/eventlet/eventlet/issues/189

Python 3.4 failure mode:
https://github.com/eventlet/eventlet/issues/476
https://github.com/eventlet/eventlet/issues/145

There are also more problems coming down the pipeline in Python 3.6:

https://github.com/eventlet/eventlet/issues/371

That one is resolved in eventlet 0.21, but we have that blocked by upper-constraints: http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n135

Zane Bitter (zaneb) wrote :

Almost certainly also affects Glance, since it appears to still use the old oslo-incubator version.

http://lists.openstack.org/pipermail/openstack-dev/2018-May/130290.html

This does indeed affect Glance. We were tracking this with https://bugs.launchpad.net/glance/+bug/1769006

Changed in glance:
status: New → Triaged
Tom Barron (tpb) on 2018-06-19
Changed in manila:
milestone: newton-1 → rocky-3
tags: added: python3
Tom Barron (tpb) on 2019-01-02
Changed in manila:
milestone: rocky-3 → stein-3
Tom Barron (tpb) on 2019-03-17
Changed in manila:
milestone: stein-3 → stein-rc1
Changed in manila:
milestone: stein-rc1 → train-1
Goutham Pacha Ravi (gouthamr) wrote :

In manila, we are no longer testing the eventlet server with SSL; we're rather testing running the eventlet server with a tls-proxy that terminates SSL requests. Distributions have been advised to switch to using manila-api behind web servers, such as apache, and most of them have done so for several releases now. I'm marking this bug to be low priority; testing and fixing it may be apt for those that really need the non-recommended/eventlet based wsgi server to work.

wiley (gfhjgfhdfjd) on 2019-06-07
summary: - requests to SSL wrapped sockets hang while reading using py3
+ Xanax without Prescription at YourRxPills.Com
description: updated
Colin Watson (cjwatson) on 2019-06-07
description: updated
summary: - Xanax without Prescription at YourRxPills.Com
+ requests to SSL wrapped sockets hang while reading using py3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.