Comment 2 for bug 1461119

Email thread
=============

Begin forwarded message:

From: Hari Prasad Killi <email address hidden>
Subject: Re: Regarding bug #1461119 (https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1461119)
Date: June 6, 2015 at 9:48:43 PM PDT
To: Chandan Mishra <email address hidden>
Cc: Megh Bhatt <email address hidden>, Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>

Ni Chandan, I was referring to the following:

Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
    data_sent += self.send(_get_memory(data, data_sent), flags)
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
    return sock.send(data, flags)
error: [Errno 32] Broken pipe
127.0.0.1 - - [02/Jun/2015 17:11:06] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 500 59
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 458, in sendall
    data_sent += self.send(_get_memory(data, data_sent), flags)
  File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 435, in send
    return sock.send(data, flags)
error: [Errno 32] Broken pipe

Regards,
Hari

From: Chandan Mishra <email address hidden>
Date: Sunday, June 7, 2015 7:33 AM
To: Hari <email address hidden>
Cc: Megh Bhatt <email address hidden>, Raj Reddy <email address hidden>, Sundaresan Rajangam <email address hidden>
Subject: Re: Regarding bug #1461119 (https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1461119)

Are you referring to:

"Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored”?
On Jun 5, 2015, at 8:13 PM, Hari Prasad Killi <email address hidden> wrote:

Hi Chandan,
I had seen this only once. There was some error backtrace in the log (showing something related to socket). Is that relevant ?

Regards,
Hari

From: Chandan Mishra <email address hidden>
Date: Saturday, June 6, 2015 6:22 AM
To: Megh Bhatt <email address hidden>
Cc: Raj Reddy <email address hidden>, Hari <email address hidden>
Subject: Re: Regarding bug #1461119 (https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1461119)

We don’t have the system in problematic stage.

@Hari: Can we reproduce this issue?

On Jun 5, 2015, at 5:50 PM, Megh Bhatt <email address hidden> wrote:

We need to check the sandesh queue. It mostly cannot get stuck in the send since send is just enqueuing on the queue but if the sandesh queue has built up due to no connection to collector or whatever reason, the message might just sit there. Is it connected to the collector? Are any other messages going?

Thanks

Megh

On Jun 5, 2015, at 5:46 PM, Chandan Mishra <email address hidden> wrote:

Megh,
Is it possible in some scenario that vrouter nodemgr gets stuck in UVE send call? Issue seen in this bug is that the vrouter nodemgr is not receiving any new events “after” this UVE send call.

@Hari: How easy is it to reproduce this problem?

Chandan

=====
Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _seqnum=0, _versionsig=2778367443, _source='nodea26', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, name='nodei30-100', deleted=False, disk_usage_info=None, process_status=None, all_core_file_list=[], _table='ObjectVRouter', process_info=[ProcessInfo(process_name='contrail-tor-agent-100', process_state='PROCESS_STATE_EXITED', last_stop_time='', start_count=36, core_file_list=[], last_start_time='1433249121234853', stop_count=0, last_exit_time='1433249125301509', exit_count=36)], description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1433249125307919, _client_context='', _connect_to_collector=True, _role=0)
=====