contrail-database-nodemgr systemd unit needs to allow access to /var/log/cassandra

Bug #1747915 reported by Robert Ayres on 2018-02-07
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Sundaresan Rajangam
R4.1
Fix Committed
High
Sundaresan Rajangam
Trunk
Invalid
High
Sundaresan Rajangam

Bug Description

Tested against Contrail 4.0.2.0-35.

Within the analyticsdb container, journald contains 1 minute cyclical error messages of a read-only filesystem for the contrail-database-nodemgr service:

root@vm01(analyticsdb):/# journalctl -u contrail-database-nodemgr
-- Logs begin at Mon 2018-01-29 16:25:59 UTC, end at Wed 2018-02-07 12:10:12 UTC. --
Jan 29 16:28:45 vm01 systemd[1]: Started "Contrail Database Node Mgr".
Jan 29 16:28:47 vm01 contrail-nodemgr[2678]: 01/29/2018 04:28:47 PM [contrail-database-nodemgr]: SANDESH: CONNECT TO COLLECTOR: True
Jan 29 16:28:48 vm01 contrail-nodemgr[2678]: 01/29/2018 04:28:48 PM [contrail-database-nodemgr]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE]
Jan 29 16:29:06 vm01 contrail-nodemgr[2678]: [Introspect:8103]127.0.0.1 - - [2018-01-29 16:29:06] "GET /Snh_SandeshUVECacheReq?x=NodeStatus HTTP/1.1" 200 3873 0.011897
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:29:08 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:30:08 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: Traceback (most recent call last):
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 244, in <module>
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: main()
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 238, in main
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: setup_logging(options)
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/bin/contrail-cassandra-status", line 187, in setup_logging
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: handlers.append(logging.FileHandler(option_group.logfile))
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: StreamHandler.__init__(self, self._open())
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: stream = open(self.baseFilename, self.mode)
Jan 29 16:31:19 vm01 contrail-nodemgr[2678]: IOError: [Errno 30] Read-only file system: '/var/log/cassandra/status.log'

Altering /lib/systemd/system/contrail-database-nodemgr.service unit file to whitelist directory entry /var/log/cassandra fixes it.

e.g.

root@vm01(analyticsdb):/# cat /lib/systemd/system/contrail-database-nodemgr.service
[Unit]
Description="Contrail Database Node Mgr"
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/contrail-nodemgr --nodetype=contrail-database
PIDFile=/var/run/contrail/contrail-database-nodemgr.pid
TimeoutStopSec=0
Restart=always
ExecStop=/bin/kill -s TERM $MAINPID
RestartPreventExitStatus=0
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/log/contrail
ReadWriteDirectories=-/etc/contrail
ReadWriteDirectories=-/var/log/cassandra

[Install]
WantedBy=multi-user.target

information type: Proprietary → Public
Jeba Paulaiyan (jebap) on 2018-03-13
tags: added: analytics
tags: added: dt
Sundaresan Rajangam (srajanga) wrote :

The patchset for https://bugs.launchpad.net/juniperopenstack/+bug/1699603 [http://github.com/Juniper/contrail-controller/commit/0c2ffd9517d393306fd2665256bd09f4561361f1] changed /var/log/cassandra/status.log to /var/log/contrail/cassandra-status.log.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers