[R4.0 ubuntu14 mitaka]contrail-vrouter-nodemgr going to inactive after vrouter -agent core dump

[aswani kumar]

R4.0 ubuntu14 mitaka build 69 openstackHA

Contrail-vrouter-nodemgr going to inactive and not able to come up after vrouter core-dump
root@nodem9:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active
contrail-vrouter-nodemgr failed

========Run time service failures=============
/var/crashes/core.contrail-vroute.3044.nodem9.1504797727

supervisord-vrouter.log
2017-09-07 20:00:35,447 INFO spawned: 'contrail-vrouter-agent' with pid 3044
2017-09-07 20:00:36,603 INFO success: contrail-vrouter-nodemgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-09-07 20:00:41,104 INFO success: contrail-vrouter-agent entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2017-09-07 20:52:07,559 INFO exited: contrail-vrouter-agent (terminated by SIGABRT (core dumped); not expected)
2017-09-07 20:52:08,431 INFO spawned: 'contrail-vrouter-agent' with pid 25712
2017-09-07 20:52:13,540 INFO success: contrail-vrouter-agent entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2017-09-07 20:52:36,926 INFO exited: contrail-vrouter-nodemgr (exit status 1; not expected)
2017-09-07 20:52:37,930 INFO spawned: 'contrail-vrouter-nodemgr' with pid 26282
2017-09-07 20:52:38,430 INFO exited: contrail-vrouter-nodemgr (exit status 1; not expected)
2017-09-07 20:52:39,433 INFO spawned: 'contrail-vrouter-nodemgr' with pid 26354
2017-09-07 20:52:39,960 INFO exited: contrail-vrouter-nodemgr (exit status 1; not expected)
2017-09-07 20:52:41,964 INFO spawned: 'contrail-vrouter-nodemgr' with pid 26414
2017-09-07 20:52:42,566 INFO exited: contrail-vrouter-nodemgr (exit status 1; not expected)
2017-09-07 20:52:45,572 INFO spawned: 'contrail-vrouter-nodemgr' with pid 26472
2017-09-07 20:52:46,091 INFO exited: contrail-vrouter-nodemgr (exit status 1; not expected)
2017-09-07 20:52:47,093 INFO gave up: contrail-vrouter-nodemgr entered FATAL state, too many start retries too quickly

/var/log/contrail-vrouter-nodemgr-stderr.log
09/07/2017 08:52:46 PM [contrail-vrouter-nodemgr]: send_process_state_db_base: Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _versionsig=2524127670, _disable_sending_object_logs=False, _seqnum=0, _source='nodem9', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, name='nodem9', installed_package_version=None, deleted=False, disk_usage_info=None, build_info='{"build-info" : [{"build-version" : "4.0.1.0", "build-time" : "2017-09-06 02:05:22.205444", "build-user" : "contrail-builder", "build-hostname" : "CB-R4-0-u14-mitaka-10-84-35-244", "build-id" : "4.0.1.0-69", "build-number" : "69"}]}', running_package_version=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=None, process_status=None, all_core_file_list=None, system_cpu_usage=None, _table='ObjectVRouter', process_info=[ProcessInfo(process_name='contrail-vrouter-agent', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1504797728000000', stop_count=0, last_exit_time='', exit_count=0), ProcessInfo(process_name='contrail-vrouter-nodemgr', process_state='PROCESS_STATE_STARTING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1504797765000000', stop_count=0, last_exit_time='', exit_count=0)], description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1504797766055681, _client_context='', _connect_to_collector=True, _disable_sending_all_messages=False, _role=0)
Traceback (most recent call last):
  File "/usr/bin/contrail-nodemgr", line 9, in <module>
    load_entry_point('nodemgr==0.1dev', 'console_scripts', 'contrail-nodemgr')()
  File "/usr/lib/python2.7/dist-packages/nodemgr/main.py", line 238, in main
    gevent.spawn(prog.run_periodically(prog.do_periodic_events, 60))])
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 1024, in run_periodically
    function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nodemgr/vrouter_nodemgr/vrouter_event_manager.py", line 41, in do_periodic_events
    self.event_tick_60()
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 980, in event_tick_60
    self.send_process_state_db(['default'])
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 663, in send_process_state_db
    group_names, ProcessInfo)
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 627, in send_process_state_db_base
    for key in self.process_state_db[group]:
KeyError: 'default'

[aswani kumar (aswanikumar90)]

copied logs to below location

stack@10.204.216.49:/cs-shared/bugs/1715706/logs
password:stack@123

[stack@anamika logs]$ ls
contrail-lbaas-haproxy-stdout.log contrail-vrouter-agent-stdout.log contrail-vrouter-nodemgr-stdout.log supervisord-vrouter.log
contrail-vrouter-agent.log contrail-vrouter-nodemgr-stderr.log contrail_vrouter_provisioning.log