Mistral greenio broken pipe when used with HAProxy

Bug #1636778 reported by Saad Zaher
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Medium
Unassigned

Bug Description

I was trying to use mistral in HA mode with haproxy and 3 running instances from mistral-server with [api, engine, executor] and I got the following error on all nodes !

anyidea what can be the problem ?

my haproxy configurations are:

listen mistral-extapi-8989
 option httpchk GET /
 bind test-cp1-vip-public-KEY-API-extapi:8989 ssl crt /etc/ssl/private//my-public-cert
 server test-cp1-c1-m1-mgmt-mistral-8989 spc-cp1-c1-m1-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m2-mgmt-mistral-8989 spc-cp1-c1-m2-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m3-mgmt-mistral-8989 spc-cp1-c1-m3-mgmt:8989 check inter 2000 rise 2 fall 5
listen mistral-mgmt-8989
 option httpchk GET /
 bind test-cp1-vip-KEY-API-mgmt:8989
 server test-cp1-c1-m1-mgmt-mistral-8989 spc-cp1-c1-m1-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m2-mgmt-mistral-8989 spc-cp1-c1-m2-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m3-mgmt-mistral-8989 spc-cp1-c1-m3-mgmt:8989 check inter 2000 rise 2 fall 5

Rabbit Conf

rabbit_hosts = test-cp1-c1-m1-mgmt:5672,test-cp1-c1-m2-mgmt:5672,test-cp1-c1-m3-mgmt:5672
rabbit_password = hMzFEQMxA4M
rabbit_use_ssl = False
rabbit_userid = mistral

Mistral Workflow Service, version 2.0.1

Launching server components [engine,api,executor]...
Server started.
2016-10-26 10:38:52.723 51383 INFO mistral.cmd.launch [-] Mistral API is serving on http://172.16.110.112:8989 (PID=51383)
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 197, in send_preamble
    self._write('Server: %s\r\n' % self.server_software)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53112)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53214)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.110 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.110 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 271, in send_headers
    self._write(str(self.headers))
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.111', 47554)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 271, in send_headers
    self._write(str(self.headers))
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53372)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53396)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

Revision history for this message
Saad Zaher (szaher) wrote :

I got this problem in stable/mitaka branch

Changed in mistral:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers