Mistral

Mistral greenio broken pipe when used with HAProxy

Bug #1636778 reported by Saad Zaher
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Triaged
Medium
Unassigned

Bug Description

I was trying to use mistral in HA mode with haproxy and 3 running instances from mistral-server with [api, engine, executor] and I got the following error on all nodes !

anyidea what can be the problem ?

my haproxy configurations are:

listen mistral-extapi-8989
 option httpchk GET /
 bind test-cp1-vip-public-KEY-API-extapi:8989 ssl crt /etc/ssl/private//my-public-cert
 server test-cp1-c1-m1-mgmt-mistral-8989 spc-cp1-c1-m1-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m2-mgmt-mistral-8989 spc-cp1-c1-m2-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m3-mgmt-mistral-8989 spc-cp1-c1-m3-mgmt:8989 check inter 2000 rise 2 fall 5
listen mistral-mgmt-8989
 option httpchk GET /
 bind test-cp1-vip-KEY-API-mgmt:8989
 server test-cp1-c1-m1-mgmt-mistral-8989 spc-cp1-c1-m1-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m2-mgmt-mistral-8989 spc-cp1-c1-m2-mgmt:8989 check inter 2000 rise 2 fall 5
 server test-cp1-c1-m3-mgmt-mistral-8989 spc-cp1-c1-m3-mgmt:8989 check inter 2000 rise 2 fall 5

Rabbit Conf

rabbit_hosts = test-cp1-c1-m1-mgmt:5672,test-cp1-c1-m2-mgmt:5672,test-cp1-c1-m3-mgmt:5672
rabbit_password = hMzFEQMxA4M
rabbit_use_ssl = False
rabbit_userid = mistral

Mistral Workflow Service, version 2.0.1

Launching server components [engine,api,executor]...
Server started.
2016-10-26 10:38:52.723 51383 INFO mistral.cmd.launch [-] Mistral API is serving on http://172.16.110.112:8989 (PID=51383)
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 197, in send_preamble
    self._write('Server: %s\r\n' % self.server_software)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53112)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:53] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53214)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.110 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.110 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 271, in send_headers
    self._write(str(self.headers))
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.111 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.111', 47554)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 271, in send_headers
    self._write(str(self.headers))
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53372)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 500 59
----------------------------------------
Exception happened during processing of request from ('172.16.110.112', 53396)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe
----------------------------------------
172.16.110.112 - - [26/Oct/2016 10:38:54] "GET / HTTP/1.0" 200 112
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 385, in sendall
    tail = self.send(data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 379, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stack/venv/mistral-20161024T143435Z/lib/python2.7/site-packages/eventlet/greenio/base.py", line 366, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

Revision history for this message
Saad Zaher (szaher) wrote :

I got this problem in stable/mitaka branch

Adriano Petrich (apetrich)
Changed in mistral:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.