[3.0-2572 icehouse] nodemgr service EXITED after installing 2572

[Ganesha HV]

Topology
========
nodeb12

Release
======
3.0-2572(ubuntu-12.04 icehouse)

After installation, I see the following status:
rootÉnodeb12:/var/log/contrail# contrail-status ö grep EXIT
contrail-vrouter-nodemgr EXITED
contrail-control-nodemgr EXITED
contrail-analytics-nodemgr EXITED
contrail-config-nodemgr EXITED
contrail-database-nodemgr EXITED

contrail-vrouter-nodemgr-stderr.log shows the following Traceback:

process:contrail-vrouter-agent,groupname:contrail-vrouter-agent,eventname:PROCESS_STATE_RUNNING
Sending UVE:NodeStatusUVE(_context='', _scope='', _category='', _send_queue_enabled=True, _seqnum=0, _versionsig=2778367443, _source='nodeb12', _instance_id='0', _client=None, _type=6, _hints=1, _http_server=None, _logger=None, _more=False, _node_type='Compute', data=NodeStatus(status=None, name='nodeb12', deleted=None, disk_usage_info=None, process_status=None, all_core_file_list=ÄÅ, _table='ObjectVRouter', process_info=ÄProcessInfo(process_name='contrail-vrouter-agent', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=2, core_file_list=ÄÅ, last_start_time='1428829727229009', stop_count=0, last_exit_time='', exit_count=0), ProcessInfo(process_name='contrail-vrouter-nodemgr', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=2, core_file_list=ÄÅ, last_start_time='1428829725123483', stop_count=0, last_exit_time='', exit_count=0), ProcessInfo(process_name='openstack-nova-compute', process_state='PROCESS_STATE_RUNNING', last_stop_time='', start_count=1, core_file_list=ÄÅ, last_start_time='1428829718592835', stop_count=0, last_exit_time='', exit_count=0)Å, description=None), _module='contrail-vrouter-nodemgr', _level=2147483647, _timestamp=1428829727230295, _client_context='', _connect_to_collector=True, _role=0)04/12/2015 02:38:47 PM Änodeb12:contrail-vrouter-nodemgr:Compute:0Å: Processing eventÄEvSandeshUVESendÅ in stateÄDisconnectÅ
04/12/2015 02:38:47 PM Änodeb12:contrail-vrouter-nodemgr:Compute:0Å: Discarding eventÄEvSandeshUVESendÅ in stateÄDisconnectÅ
wokeup and found a line
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Warning keystonerc not sourced
Openstack Nova Compute status unchanged at:PROCESS_STATE_RUNNING
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/gevent/greenlet.py", line 328, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/nodemgr/VrouterNodemgr.py", line 143, in runforever
    prev_current_time = self.event_tick_60(prev_current_time)
  File "/usr/lib/python2.7/dist-packages/nodemgr/EventManager.py", line 334, in event_tick_60
    self.send_disk_usage_info()
  File "/usr/lib/python2.7/dist-packages/nodemgr/VrouterNodemgr.py", line 93, in send_disk_usage_info
    self.send_disk_usage_info_base(NodeStatusUVE, NodeStatus, DiskPartitionUsageStats)
  File "/usr/lib/python2.7/dist-packages/nodemgr/EventManager.py", line 250, in send_disk_usage_info_base
    partition = subprocess.Popen("df -T -t ext2 -t ext3 -t ext4 -t xfs",
NameError: global name 'subprocess' is not defined
<Greenlet at 0x306c050: <bound method VrouterEventManager.runforever of <nodemgr.VrouterNodemgr.VrouterEventManager instance at 0x2ca0638>>> failed with NameError

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9063
Submitter: Shabbir Hussain (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9063
Committed: http://github.org/Juniper/contrail-controller/commit/be7553ef34350a405c2b242385db3efb99ea1ea2
Submitter: Zuul
Branch: master

commit be7553ef34350a405c2b242385db3efb99ea1ea2
Author: Shabbir Hussain <email address hidden>
Date: Sun Apr 12 23:49:58 2015 -0700

Nodemgr service EXITED for all nodes due to import error.

Fix: Import subprocess module in EventManager module.

Closes-Bug: #1443049

Change-Id: Iffbafba9217f701b7b842e2fd080df81ec4a8105

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9155
Submitter: Shabbir Hussain (<email address hidden>)

[Abhay Joshi (abhayj)]

I would prefer that you didn't merge this

When contrail mainline builds (3.0 build 2573) is provisioned using server manager, the node manager configuration files are not created correctly. This change in node manager configuration has been added in mainline, but corresponding changes to contrail-puppet repo to create these configuration files are missing. We need to add the templates for these config files and also modify the existing puppet manifests to create the config files on target nodes using the templates.

Assigning the bug to person who has done the original code change in contrail and fab provisioning. SM manager will provide help in implementing, but would like the person doing original change take care of this.

Please take care of SM changes also to complete the function. SM team can assist.

[Bharat Kumar (pbharat)]

With Server-manager its in falied state

root@nodeg37:~# contrail-status | grep failed
contrail-control-nodemgr failed
contrail-analytics-nodemgr failed
contrail-config-nodemgr failed
contrail-database-nodemgr failed
root@nodeg37:~#

[Bharat Kumar (pbharat)]

Tested on R2.20 Build #1

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/9155
Submitter: Raj Reddy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/9155
Committed: http://github.org/Juniper/contrail-controller/commit/abfd62dac5c8ef426de736526a128bb992d67c64
Submitter: Zuul
Branch: R2.20

commit abfd62dac5c8ef426de736526a128bb992d67c64
Author: Shabbir Hussain <email address hidden>
Date: Sun Apr 12 23:49:58 2015 -0700

Nodemgr service EXITED for all nodes due to import error.

Fix: Import subprocess module in EventManager module.

Closes-Bug: #1443049

Change-Id: Iffbafba9217f701b7b842e2fd080df81ec4a8105