In latest CB mainline build 18, contrail-vrouter-nodemgr is not coming to active state. If we restart the service, could see below error messages in contrail-vrouter-nodemgr-stderr.log.
Issue seen in all compute nodes (nodel1 and nodei10)
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: CONNECT TO COLLECTOR: True
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_INFO]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: Logging: FILE: [None] -> [<stdout>]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: Trace: PRINT: [None] -> [False]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: Flow Logging: [None] -> [False]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Cannot write http_port 8102 to /tmp/contrail-vrouter-nodemgr.2608.http_port
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Starting Introspect on HTTP Port 8102
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Processing event[EvStart] in state[none]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Processing event[EvSandeshUVESend] in state[Idle]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei10 process_status = [ << module_id = contrail-vrouter-nodemgr instance_id =
0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down
>>, ] >>
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Discarding event[EvSandeshUVESend] in state[Idle]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Processing event[EvSandeshUVESend] in state[Idle]
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei10:Compute:contrail-vrouter-nodemgr:0 client_info = << status = Id
le successful_connections = 0 pid = 4316 http_port = 8102 start_time = 1502443648912681 collector_name = collector_ip = collector_list = [ 22.22.22.55:8086, 22.22.22.56:8086, ] >> sm_queue_cou
nt = 1 max_sm_queue_count = 1 >>
08/11/2017 02:57:28 PM [contrail-vrouter-nodemgr]: Discarding event[EvSandeshUVESend] in state[Idle]
08/11/2017 02:57:29 PM [contrail-vrouter-nodemgr]: send_nodemgr_process_status_base: Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _versionsig=2524127670, _disa
ble_sending_object_logs=False, _seqnum=0, _source='nodei10', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=N
one, name='nodei10', installed_package_version=None, deleted=None, disk_usage_info=None, build_info=None, running_package_version=None, process_mem_cpu_usage=None, system_cpu_info=None, system_mem_usage=N
one, process_status=[ProcessStatus(instance_id='0', module_id='contrail-vrouter-nodemgr', state='Functional', description='', connection_infos=None)], all_core_file_list=None, system_cpu_usage=None, _tabl
e='ObjectVRouter', process_info=None, description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1502443649125067, _client_context='', _connect_to_collector=True, _disable_sendin
g_all_messages=False, _role=0)
08/11/2017 02:57:29 PM [contrail-vrouter-nodemgr]: send_process_state_db_base: Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _versionsig=2524127670, _disable_se
nding_object_logs=False, _seqnum=0, _source='nodei10', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, n
ame='nodei10', installed_package_version=None, deleted=False, disk_usage_info=None, build_info='{"build-info" : [{"build-version" : "4.1.0.0", "build-time" : "2017-08-10 15:01:33.882549", "build-user" : "
contrail-builder", "build-hostname" : "CB-mainline-ubuntu14-10-84-35-212", "build-id" : "4.1.0.0-18", "build-number" : "18"}]}', running_package_version=None, process_mem_cpu_usage=None, system_cpu_info=N
one, system_mem_usage=None, process_status=None, all_core_file_list=None, system_cpu_usage=None, _table='ObjectVRouter', process_info=[ProcessInfo(process_name='contrail-vrouter-agent', process_state='PRO
CESS_STATE_RUNNING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1502443093000000', stop_count=0, last_exit_time='', exit_count=0), ProcessInfo(process_name='contrail-vrouter-nod
emgr', process_state='PROCESS_STATE_STARTING', last_stop_time='', start_count=1, core_file_list=[], last_start_time='1502443648000000', stop_count=0, last_exit_time='', exit_count=0)], description=None),
_module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1502443649125801, _client_context='', _connect_to_collector=True, _disable_sending_all_messages=False, _role=0)
Traceback (most recent call last):
File "/usr/bin/contrail-nodemgr", line 9, in <module>
load_entry_point('nodemgr==0.1dev', 'console_scripts', 'contrail-nodemgr')()
File "/usr/lib/python2.7/dist-packages/nodemgr/main.py", line 237, in main
gevent.spawn(prog.run_periodically(prog.do_periodic_events, 60))])
File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 1021, in run_periodically
function(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nodemgr/vrouter_nodemgr/vrouter_event_manager.py", line 43, in do_periodic_events
self.event_tick_60()
File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 977, in event_tick_60
self.send_process_state_db(['default'])
File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 660, in send_process_state_db
group_names, ProcessInfo)
File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 624, in send_process_state_db_base
for key in self.process_state_db[group]:
KeyError: 'default'
Review in progress for https:/
Submitter: Megh Bhatt (<email address hidden>)