spawn_n fails in functional tests

Bug #1546506 reported by Jakub Libosvar
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Jakub Libosvar

Bug Description

Gate seems broken due to spawn_n failure, doesn't seem like Neutron's fault but library issue. I haven't tracked which library was updated yet. Started occurring at about Feb 16/17 midnight UTC. Also influences fullstack tests.

e-s: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22gate-neutron-dsvm-functional%5C%22%20AND%20build_status%3A%5C%22FAILURE%5C%22%20AND%20message%3A%5C%22greenpool.py%5C%5C%5C%22%2C%20line%2082%2C%20in%20_spawn_n_impl%5C%22

Revision history for this message
Jakub Libosvar (libosvar) wrote :

Traceback example:
ft2.9: neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_failover_StringException: Empty attachments:
  stdout

pythonlogging:'': {{{WARNING [oslo_config.cfg] Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.}}}
stderr: {{{
Traceback (most recent call last):
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 719, in process_request
    proto.__init__(sock, address, self)
  File "neutron/agent/linux/utils.py", line 381, in __init__
    server)
  File "/usr/lib/python2.7/SocketServer.py", line 647, in __init__
    self.setup()
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 306, in setup
    conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 95] Operation not supported
Traceback (most recent call last):
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 719, in process_request
    proto.__init__(sock, address, self)
  File "neutron/agent/linux/utils.py", line 381, in __init__
    server)
  File "/usr/lib/python2.7/SocketServer.py", line 647, in __init__
    self.setup()
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 306, in setup
    conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, True)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 95] Operation not supported
}}}

Traceback (most recent call last):
  File "neutron/tests/functional/agent/l3/test_dvr_router.py", line 654, in test_dvr_ha_router_failover
    utils.wait_until_true(lambda: router1.ha_state == 'backup')
  File "neutron/agent/linux/utils.py", line 326, in wait_until_true
    eventlet.sleep(sleep)
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 34, in sleep
    hub.switch()
  File "/opt/stack/new/neutron/.tox/dsvm-functional-constraints/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

Revision history for this message
Jakub Libosvar (libosvar) wrote :

The culprit it eventlet 0.18.3 - 0.18.2 doesn't reproduce the issue.

Henry Gessau (gessau)
Changed in neutron:
importance: Undecided → Critical
status: New → Confirmed
Revision history for this message
Victor Stinner (vstinner) wrote :

I opened a bug report in eventlet: https://github.com/eventlet/eventlet/issues/301

You may try to workaround the issue (until it is fixed) by setting disable_nagle_algorithm = False on the WSGI server.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/281278

Changed in neutron:
assignee: nobody → Jakub Libosvar (libosvar)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/281278
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=21d139d441f9e43b35b2b96d05e962dff9a690a1
Submitter: Jenkins
Branch: master

commit 21d139d441f9e43b35b2b96d05e962dff9a690a1
Author: Jakub Libosvar <email address hidden>
Date: Wed Feb 17 13:35:25 2016 +0000

    Don't disable Nagle algorithm in HttpProtocol

    0.18.3 evenlet by default disables Nagle algorithm for sockets using
    http protocol [1]. It brought a regression to eventlet [2] and this
    patch adds workaround that doesn't disable the algorithm.

    [1] https://github.com/eventlet/eventlet/commit/40714b1ffadd47b315ca07f9b85009448f0fe63d
    [2] https://github.com/eventlet/eventlet/issues/301

    Change-Id: I79a8583a5fe9812b6609bd4df5623f13c3b81df5
    Closes-bug: 1546506

Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers