Upgrade from R2.1-54 to R2.2-17 using SM renders vrouter nodemgr to EXITED and NTP un-synchronized.

[Ritam Gangopadhyay]

Steps:
1. Using server-manager 14.04 ubuntu R2.2 - build 17 juno I reimaged and provisioned a single node setup to 14.04 ubuntu R2.1 icehouse build 54. Done using sequence_provisioning set to false.
2. Upgraded the node, sequence_provisioning set to false, using 14.04 ubuntu icehouse R2.2 build 17.

After provisioning went through I see the status of the daemons as below:

root@nodec14:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active
contrail-vrouter-nodemgr EXITED

== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr initializing (NTP state unsynchronized.)
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr initializing (NTP state unsynchronized.)
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology failed

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr initializing (NTP state unsynchronized.)
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr initializing (NTP state unsynchronized.)
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@nodec14:~#

root@nodec14:~# cat /var/log/contrail/contrail-vrouter-nodemgr-stderr.log

SNIP ###############

Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _seqnum=0, _versionsig=2778367443, _source='nodec14', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, name='nodec14', deleted=None, disk_usage_info=[DiskPartitionUsageStats(partition_space_available_1k=419373220, partition_space_used_1k=5176840, partition_name='/dev/mapper/nodec14--vg-root', partition_type='ext4'), DiskPartitionUsageStats(partition_space_available_1k=162172, partition_space_used_1k=66359, partition_name='/dev/sda1', partition_type='ext2')], process_status=None, all_core_file_list=None, _table='ObjectVRouter', process_info=None, description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1431418409924085, _client_context='', _connect_to_collector=True, _role=0)Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/nodemgr/VrouterNodemgr.py", line 162, in runforever
    lb_stats.send_loadbalancer_stats()
  File "/usr/lib/python2.7/dist-packages/nodemgr/LoadbalancerStats.py", line 50, in send_loadbalancer_stats
    _send_loadbalancer_uve(driver)
  File "/usr/lib/python2.7/dist-packages/nodemgr/LoadbalancerStats.py", line 35, in _send_loadbalancer_uve
    for pool_uuid in os.listdir(LB_BASE_DIR):
OSError: [Errno 2] No such file or directory: '/var/lib/contrail/loadbalancer/'
<Greenlet at 0x7fa3c99fa550: <bound method VrouterEventManager.runforever of <nodemgr.VrouterNodemgr.VrouterEventManager object at 0x7fa3c9fbf690>>> failed with OSError

05/12/2015 01:13:30 AM [nodec14:contrail-vrouter-nodemgr:Compute:0]: Processing event[EvSandeshUVESend] in state[Disconnect]
05/12/2015 01:13:30 AM [nodec14:contrail-vrouter-nodemgr:Compute:0]: Discarding event[EvSandeshUVESend] in state[Disconnect]

SNIP ###############

[Abhay Joshi (abhayj)]

The node manager status might not be due to upgrade. There is an issuw with Fresh provision to with node manager statuses. Shabbir is working on this. Let's please check with him first.

[Abhay Joshi (abhayj)]

Confirmed with Thilak too. I had seen this on fresh provision and so did Thilak. We need to have a new bug opened to track node manager statuses.

Also, NTP un-synchronized is not related to upgrade. Will ask Thilak to confirm and close this bug. We should have bugs to track node-manager status and NTP synchronization (not due to upgrade).

[Thilak Raj (tsurendra)]

This is not a bug caused by upgrade.
Please open a new Bug.