Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.write(data)
File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
self.send_headers()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
'Date: %s\r\n' % format_date_time(time.time())
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
127.0.0.1 - - [02/Jun/2015 17:11:06] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 500 59
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
self.wfile.close()
File "/usr/lib/python2.7/socket.py", line 279, in close
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
Regards,
Hari
From: Chandan Mishra <email address hidden>
Date: Sunday, June 7, 2015 7:33 AM
To: Hari <email address hidden>
Cc: Megh Bhatt <email address hidden>, Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>
Subject: Re: Regarding bug #1461119 (https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1461119)
Are you referring to:
"Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored”?
On Jun 5, 2015, at 8:13 PM, Hari Prasad Killi <email address hidden> wrote:
Hi Chandan,
I had seen this only once. There was some error backtrace in the log (showing something related to socket). Is that relevant ?
On Jun 5, 2015, at 5:50 PM, Megh Bhatt <email address hidden> wrote:
We need to check the sandesh queue. It mostly cannot get stuck in the send since send is just enqueuing on the queue but if the sandesh queue has built up due to no connection to collector or whatever reason, the message might just sit there. Is it connected to the collector? Are any other messages going?
Thanks
Megh
On Jun 5, 2015, at 5:46 PM, Chandan Mishra <email address hidden> wrote:
Megh,
Is it possible in some scenario that vrouter nodemgr gets stuck in UVE send call? Issue seen in this bug is that the vrouter nodemgr is not receiving any new events “after” this UVE send call.
Email thread
=============
Begin forwarded message:
From: Hari Prasad Killi <email address hidden> /bugs.launchpad .net/juniperope nstack/ r2.20/+ bug/1461119)
Subject: Re: Regarding bug #1461119 (https:/
Date: June 6, 2015 at 9:48:43 PM PDT
To: Chandan Mishra <email address hidden>
Cc: Megh Bhatt <email address hidden>, Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>
Ni Chandan, I was referring to the following:
Traceback (most recent call last): python2. 7/wsgiref/ handlers. py", line 86, in run finish_ response( ) python2. 7/wsgiref/ handlers. py", line 128, in finish_response write(data) python2. 7/wsgiref/ handlers. py", line 212, in write send_headers( ) python2. 7/wsgiref/ handlers. py", line 270, in send_headers send_preamble( ) python2. 7/wsgiref/ handlers. py", line 194, in send_preamble date_time( time.time( )) python2. 7/socket. py", line 324, in write python2. 7/socket. py", line 303, in flush _sock.sendall( view[write_ offset: write_offset+ buffer_ size]) python2. 7/dist- packages/ gevent/ socket. py", line 458, in sendall _get_memory( data, data_sent), flags) python2. 7/dist- packages/ gevent/ socket. py", line 435, in send CacheReq? x=NodeStatus HTTP/1.1" 500 59 python2. 7/SocketServer. py", line 295, in _handle_ request_ noblock process_ request( request, client_address) python2. 7/SocketServer. py", line 321, in process_request finish_ request( request, client_address) python2. 7/SocketServer. py", line 334, in finish_request RequestHandlerC lass(request, client_address, self) python2. 7/SocketServer. py", line 651, in __init__ python2. 7/SocketServer. py", line 710, in finish wfile.close( ) python2. 7/socket. py", line 279, in close python2. 7/socket. py", line 303, in flush _sock.sendall( view[write_ offset: write_offset+ buffer_ size]) python2. 7/dist- packages/ gevent/ socket. py", line 458, in sendall _get_memory( data, data_sent), flags) python2. 7/dist- packages/ gevent/ socket. py", line 435, in send
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
'Date: %s\r\n' % format_
File "/usr/lib/
self.flush()
File "/usr/lib/
self.
File "/usr/lib/
data_sent += self.send(
File "/usr/lib/
return sock.send(data, flags)
error: [Errno 32] Broken pipe
127.0.0.1 - - [02/Jun/2015 17:11:06] "GET /Snh_SandeshUVE
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.finish()
File "/usr/lib/
self.
File "/usr/lib/
self.flush()
File "/usr/lib/
self.
File "/usr/lib/
data_sent += self.send(
File "/usr/lib/
return sock.send(data, flags)
error: [Errno 32] Broken pipe
Regards,
Hari
From: Chandan Mishra <email address hidden> /bugs.launchpad .net/juniperope nstack/ r2.20/+ bug/1461119)
Date: Sunday, June 7, 2015 7:33 AM
To: Hari <email address hidden>
Cc: Megh Bhatt <email address hidden>, Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>
Subject: Re: Regarding bug #1461119 (https:/
Are you referring to:
"Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/ python2. 7/threading. pyc'> ignored”?
On Jun 5, 2015, at 8:13 PM, Hari Prasad Killi <email address hidden> wrote:
Hi Chandan,
I had seen this only once. There was some error backtrace in the log (showing something related to socket). Is that relevant ?
Regards,
Hari
From: Chandan Mishra <email address hidden> /bugs.launchpad .net/juniperope nstack/ r2.20/+ bug/1461119)
Date: Saturday, June 6, 2015 6:22 AM
To: Megh Bhatt <email address hidden>
Cc: Raj Reddy <email address hidden>, Hari <email address hidden>
Subject: Re: Regarding bug #1461119 (https:/
We don’t have the system in problematic stage.
@Hari: Can we reproduce this issue?
On Jun 5, 2015, at 5:50 PM, Megh Bhatt <email address hidden> wrote:
We need to check the sandesh queue. It mostly cannot get stuck in the send since send is just enqueuing on the queue but if the sandesh queue has built up due to no connection to collector or whatever reason, the message might just sit there. Is it connected to the collector? Are any other messages going?
Thanks
Megh
On Jun 5, 2015, at 5:46 PM, Chandan Mishra <email address hidden> wrote:
Megh,
Is it possible in some scenario that vrouter nodemgr gets stuck in UVE send call? Issue seen in this bug is that the vrouter nodemgr is not receiving any new events “after” this UVE send call.
@Hari: How easy is it to reproduce this problem?
Chandan
===== VE(_context= '', _scope='', _category='', _send_queue_ enabled= True, _seqnum=0, _versionsig= 2778367443, _source='nodea26', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type= 'Compute' , data=NodeStatus (status= None, name='nodei30-100', deleted=False, disk_usage_ info=None, process_ status= None, all_core_ file_list= [], _table= 'ObjectVRouter' , process_ info=[ProcessIn fo(process_ name='contrail- tor-agent- 100', process_ state=' PROCESS_ STATE_EXITED' , last_stop_time='', start_count=36, core_file_list=[], last_start_ time='143324912 1234853' , stop_count=0, last_exit_ time='143324912 5301509' , exit_count=36)], description=None), _module= 'contrail- vrouter- nodemgr' , _level=2147483647, _timestamp= 143324912530791 9, _client_context='', _connect_ to_collector= True, _role=0)
Sending UVE:NodeStatusU
=====