CAN upgrade - 4.0.3.0-38 to 4.1.1.0-128: contrail-database-nodemgr and contrail-analytics-nodemgr fails

[Rajeshwari Srinivasan]

After upgrade of CAN, contrail-database-nodemgr and contrail-analytics-nodemgr are in failed state.
We haven’t changed any of our scripts in CSO. We have only changed the version from 4.0.0-9 to 4.0.0-128.
We didn’t notice this issue during upgrade from 4.0.3.0-38 to 4.0.0-9.

oot@ibmflex-esx2-vm2(analyticsdb):/# tail -F /var/log/contrail/contrail-database-nodemgr-stderr.log
    backupCount=backupCount))
  File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/contrail/contrail-database-nodemgr.log'

Traceback (most recent call last):
  File "/usr/bin/contrail-nodemgr", line 9, in <module>
    load_entry_point('nodemgr==0.1dev', 'console_scripts', 'contrail-nodemgr')()
  File "/usr/lib/python2.7/dist-packages/nodemgr/main.py", line 191, in main
    prog = AnalyticsEventManager(_args, rule_file, unit_names)
  File "/usr/lib/python2.7/dist-packages/nodemgr/analytics_nodemgr/analytics_event_manager.py", line 26, in __init__
    rule_file, sandesh_global)
  File "/usr/lib/python2.7/dist-packages/nodemgr/common/event_manager.py", line 414, in __init__
    syslog_facility=self.config.syslog_facility)
  File "/usr/lib/python2.7/dist-packages/pysandesh/sandesh_base.py", line 295, in set_logging_params
    enable_flow_log=enable_flow_log)
  File "/usr/lib/python2.7/dist-packages/pysandesh/sandesh_logger.py", line 95, in set_logging_params
    self.set_logging_file(file, maxBytes, backupCount)
  File "/usr/lib/python2.7/dist-packages/pysandesh/sandesh_logger.py", line 171, in set_logging_file
    backupCount=backupCount))
  File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/__init__.py", line 903, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.7/logging/__init__.py", line 928, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/contrail/contrail-analytics-nodemgr.log'

[Santosh Gupta (sangupta)]

R4.0 onwards there is a new log file contrail-*-nodemgr.log. The ownership of that file is not changed. https://review.opencontrail.org/#/c/43052/1/common/debian/contrail-openstack-database/debian/contrail-openstack-database.postinst
https://review.opencontrail.org/#/c/43052/1/common/rpm/contrail-openstack-database.spec
This needs to be fixed in R4.0 onwards.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48197
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48198
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48199
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48200
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/48197
Committed: http://github.com/Juniper/contrail-packages/commit/717fb495d8f1d2ea01f46133c822e388cb0c7788
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 717fb495d8f1d2ea01f46133c822e388cb0c7788
Author: Santosh Gupta <email address hidden>
Date: Thu Dec 13 13:58:40 2018 -0800

Change ownership of contrail-<>-nodemgr.log file on upgrade (1/2)

New file is introduced by default in nodemgr for logging from R4.0 onwards.
Change ownership to contrail:contrail.
- contrail-<config|control|analytics|database|vrouter>-nodemgr.log

Change-Id: I377a8d86fea4e172a1df522f9941c620f33520fd
Closes-Bug: #1777781

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/48198
Committed: http://github.com/Juniper/contrail-packaging/commit/2ceac1baf43bf380cdcac63d4c49f588546573c1
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 2ceac1baf43bf380cdcac63d4c49f588546573c1
Author: Santosh Gupta <email address hidden>
Date: Thu Dec 13 13:59:31 2018 -0800

Change ownership of contrail-<>-nodemgr.log file on upgrade (2/2)

New file is introduced by default in nodemgr for logging from R4.0 onwards.
Change ownership to contrail:contrail.
- contrail-<config|control|analytics|database|vrouter>-nodemgr.log

Change-Id: Ic6e9dce22a16e93ed20a6978fc6a2ce072d39d5e
Closes-Bug: #1777781