contrail-database-nodemgr systemd unit needs to allow access to /var/log/cassandra

[Robert Ayres]

Tested against Contrail 4.0.2.0-35.

Within the analyticsdb container, journald contains 1 minute cyclical error messages of a read-only filesystem for the contrail-database-nodemgr service:

root@vm01(analyticsdb):/# journalctl -u contrail-database-nodemgr
-- Logs begin at Mon 2018-01-29 16:25:59 UTC, end at Wed 2018-02-07 12:10:12 UTC. --
Jan 29 16:28:45 vm01 systemd[1]: Started "Contrail Database Node Mgr".
Jan 29 16:28:47 vm01 contrail-nodemgr[2678]: 01/29/2018 04:28:47 PM [contrail-database-nodemgr]: SANDESH: CONNECT TO COLLECTOR: True
Jan 29 16:28:48 vm01 contrail-nodemgr[2678]: 01/29/2018 04:28:48 PM [contrail-database-nodemgr]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE]
Jan 29 16:29:06 vm01 contrail-nodemgr[2678]: [Introspect:8103]127.0.0.1 - - [2018-01-29 16:29:06] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 3873 0.011897
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'

Altering /lib/systemd/system/contrail-database-nodemgr.service unit file to whitelist directory entry /var/log/cassandra fixes it.

e.g.

root@vm01(analyticsdb):/# cat /lib/systemd/system/contrail-database-nodemgr.service
[Unit]
Description="Contrail Database Node Mgr"
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/contrail-nodemgr --nodetype=contrail-database
PIDFile=/var/run/contrail/contrail-database-nodemgr.pid
TimeoutStopSec=0
Restart=always
ExecStop=/bin/kill -s TERM $MAINPID
RestartPreventExitStatus=0
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/log/contrail
ReadWriteDirectories=-/etc/contrail
ReadWriteDirectories=-/var/log/cassandra

[Install]
WantedBy=multi-user.target

[Sundaresan Rajangam (srajanga)]

The patchset for https://bugs.launchpad.net/juniperopenstack/+bug/1699603 [http://github.com/Juniper/contrail-controller/commit/0c2ffd9517d393306fd2665256bd09f4561361f1] changed /var/log/cassandra/status.log to /var/log/contrail/cassandra-status.log.