contrail-vrouter-nodemgr EXITED due to NoSuchProcess

[Yan Chen]

3.0.2.0-51

root@rdm3r28c008:~# contrail-status -d
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active pid 18908, uptime 15 days, 21:52:52
contrail-vrouter-nodemgr EXITED Oct 05 05:52 AM

10/05/2016 05:52:00 AM [contrail-vrouter-nodemgr]: send_process_state_db_base: Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _seqnum=0, _versionsig=24069590, _source='rdm3r28c008.rdm3.cci.att.com', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, name='rdm3r28c008.rdm3.cci.att.com', deleted=False, disk_usage_info=None, system_mem_cpu_usage=None, process_mem_cpu_usage=None, system_cpu_info=None, process_status=None, all_core_file_list=None, _table='ObjectVRouter', process_info=[ProcessInfo(process_name='contrail-vrouter-agent', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1475646720595880', stop_count=0, last_exit_time='1475646716235350', exit_count=1), ProcessInfo(process_name='contrail-vrouter-nodemgr', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1475646716252916', stop_count=0, last_exit_time='1475646716244692', exit_count=1)], description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1475646720596085, _client_context='', _connect_to_collector=True, _role=0)
wokeup and found a line
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
ls: cannot access /var/crashes: No such file or directory
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/nodemgr/vrouter_nodemgr/vrouter_event_manager.py", line 139, in runforever
    prev_current_time = self.event_tick_60(prev_current_time)
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 511, in event_tick_60
    process_mem_cpu_usage = self.get_all_processes_mem_cpu_usage()
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 396, in get_all_processes_mem_cpu_usage
    process_mem_cpu = mem_cpu_usage_data.get_process_mem_cpu_info()
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/cpuinfo.py", line 86, in get_process_mem_cpu_info
    process_mem_cpu.cpu_share = self._process.get_cpu_percent(interval=0.1)/psutil.NUM_CPUS
  File "/usr/lib/python2.7/dist-packages/psutil/__init__.py", line 709, in get_cpu_percent
    pt2 = self._platform_impl.get_cpu_times()
  File "/usr/lib/python2.7/dist-packages/psutil/_pslinux.py", line 470, in wrapper
    raise NoSuchProcess(self.pid, self._process_name)
NoSuchProcess: process no longer exists (pid=3944)
<Greenlet at 0x7f41b6c9a050: <bound method VrouterEventManager.runforever of <nodemgr.vrouter_nodemgr.vrouter_event_manager.VrouterEventManager object at 0x7f41b7d26e50>>> failed with NoSuchProcess

Process 3944 was contrail-grouter-agent. It was killed due to out of memory.

/var/log/kern.log-20161005-1475652905:<3>Oct 5 05:52:00 rdm3r28c008 kernel: [1512199.514477] Out of memory: Kill process 3944 (contrail-vroute) score 1 or sacrifice child
/var/log/kern.log-20161005-1475652905:<3>Oct 5 05:52:00 rdm3r28c008 kernel: [1512199.560521] Out of memory: Kill process 3944 (contrail-vroute) score 1 or sacrifice child
/var/log/kern.log-20161005-1475652905:<3>Oct 5 05:52:00 rdm3r28c008 kernel: [1512199.582288] Killed process 3944 (contrail-vroute) total-vm:843932kB, anon-rss:163284kB, file-rss:16012kB

[Santosh Gupta (sangupta)]

Dup of 1596867