contrail-config-manager fails to come up on Contrail upgrade (Fuel 9.1/MOS9)

[Savithru Lokanath]

I have a MOS9 setup in 3+3+3 (Contrail CFGM + Contrail Analytics + Contrail Analytics DB) config & when I upgraded the setup from Contrail 3.1.0.0-25 to Contrail 3.1.2.0-65, contrail-config-nodemgr process failed on the upgrade step.

Looks like there is a process /usr/bin/contrail-cassandra-repair which is using the contrail-device-manager service's port (8100). This is not seen on other 2 nodes.

Below are some info,

root@node-10:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr failed
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema backup
contrail-svc-monitor backup
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: inactive (disabled on boot)
unix:///var/run/supervisord_support_service.sockno

========================================================================================

Output of /var/log/contrail/contrail-config-nodemgr-stderr.log

root@node-10:~# tail -f /var/log/contrail/contrail-config-nodemgr-stderr.log
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Exception AssertionError: AssertionError() in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
Discovery server: 192.168.40.1
Discovery port: 5998
Collector address: []
02/23/2017 12:31:07 AM [contrail-config-nodemgr]: SANDESH: CONNECT TO COLLECTOR: True
02/23/2017 12:31:07 AM [contrail-config-nodemgr]: Unable to open HTTP Port 8100, [Errno 98] Address already in use

========================================================================================

root@node-10:~# netstat -alnp | grep 8100
tcp 0 0 0.0.0.0:8100 0.0.0.0:* LISTEN 20176/python

========================================================================================

root@node-10:~# ps -aux | grep 20176
root 2909 0.0 0.0 10460 928 pts/11 S+ 00:39 0:00 grep --color=auto 20176
root 20176 0.0 0.0 65908 15524 ? S Feb21 0:00 /usr/bin/python /usr/bin/contrail-cassandra-repair --log-file /var/log/contrail/repair.log --debug

[Illia Polliul (ipolliul)]

Per conversation with Savithru, this problem didn't reproduce.