Constant tracebacks with eventlet 0.18.2

[Jesse Keating]

Kilo builds, with eventlet 0.18.2 have a constant traceback:

2016-02-12 00:47:01.126 3936 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f05a0acfbd0>>' _process_stack /opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:783
2016-02-12 00:47:01.129 3936 INFO nova.osapi_compute.wsgi.server [-] Traceback (most recent call last):
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 501, in handle_one_response
    write(b''.join(towrite))
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 442, in write
    _writelines(towrite)
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/support/__init__.py", line 62, in safe_writelines
    writeall(fd, item)
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/support/__init__.py", line 67, in writeall
    fd.write(buf)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 383, in sendall
    tail = self.send(data, flags)
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/bbc/openstack-11.0-bbc153/nova/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 364, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

This is happening across nova, neutron, glance, etc..

Dropping back to eventlet < 0.18.0 works.

[Markus Zoeller (markus_z) (mzoeller)]

@Jesse Keating:

The periodic check for stable/kilo python27 shows no failure [1]
although it uses eventlet version 0.18.2 [2].

* What's the exact version of Nova you are using?
* What are the steps which lead to that traceback?

Please switch the status back to "new" after you provided the needed
information.

References:
[1] http://logs.openstack.org/periodic-stable/periodic-nova-python27-kilo/0015699/testr_results.html.gz
[2] http://logs.openstack.org/periodic-stable/periodic-nova-python27-kilo/0015699/console.html#_2016-02-12_07_12_51_249

[Markus Zoeller (markus_z) (mzoeller)]

I double-checked in a local VM with Devstack if I can create an Instance with latest "stable/kilo" code and it worked.

Nova version:
    $ git log --no-merges --oneline -1
    d95b321 Replace eventlet-based raw socket client with requests

Eventlet version:
    $ pip show eventlet
    ---
    Metadata-Version: 2.0
    Name: eventlet
    Version: 0.18.2
    Summary: Highly concurrent networking library
    Home-page: http://eventlet.net
    Author: Linden Lab
    Author-email: <email address hidden>
    License: UNKNOWN
    Location: /usr/local/lib/python2.7/dist-packages
    Requires: greenlet

[Matt Riedemann (mriedem)]

Here is a heat stable/kilo job with the traceback in the n-api logs:

http://logs.openstack.org/66/278466/4/check/gate-heat-dsvm-functional-orig-mysql/faabdda/logs/screen-n-api.txt.gz?level=INFO#_2016-02-11_23_15_46_870

[Matt Riedemann (mriedem)]

FWIW, I don't think it's just kilo:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Traceback%20(most%20recent%20call%20last)%3A%5C%22%20AND%20message%3A%5C%22handle_one_response%5C%22%20AND%20module%3A%5C%22nova.osapi_compute.wsgi.server%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

[Matt Riedemann (mriedem)]

This is actually better, it shows the hits in the other service API logs:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Traceback%20(most%20recent%20call%20last)%3A%5C%22%20AND%20message%3A%5C%22handle_one_response%5C%22%20AND%20tags%3A%5C%22screen%5C%22&from=7d

[Matt Riedemann (mriedem)]

Someone should probably open an issue against eventlet. This was the change:

https://github.com/eventlet/eventlet/commit/b7380fdc70fde26777f4f141da3f01570e870cac

[Jesse Keating (jesse-keating)]

I've filed an eventlet bug at https://github.com/eventlet/eventlet/issues/299

[Matt Riedemann (mriedem)]

eventlet 0.18.3 is out. We probably need some more time to keep an eye on this query to see if things drop off, then we'll know if it's fixed.

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Traceback%20(most%20recent%20call%20last):%5C%22%20AND%20message:%5C%22handle_one_response%5C%22%20AND%20tags:%5C%22screen%5C%22&from=7d

[Davanum Srinivas (DIMS) (dims-v)]

Looks like this dropped off

[Armando Migliaccio (armando-migliaccio)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Cyril Roelandt (cyril-roelandt)]

I don't think this has been an issue in recent versions of Glance/Python/Eventlet. I'll close this bug, feel free to reopen if needed.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]