Juniper Openstack

Openshift : [Build 4.0.1.0-32] contrail-kube-manager failed to come up due to log permission issue

  1. Series r4.0
  2. Bug #1719430
Bug #1719430 reported by chhandak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
R4.0
Fix Committed
High
Aniket Gawade
Juniper Openstack r4.0.3.0
R4.1
Fix Committed
High
Aniket Gawade
Juniper Openstack r4.1.1.0
Trunk
Won't Fix
High
Aniket Gawade
Juniper Openstack r5.0.0

Bug Description

While upgrading Openshift cluster from build 31 to 32 in Redhat base os, contrail kube-manager failed to come up for pemission issue with contrail-kube-manager.log

What Has been tried
---------------------
The cluster was up with Build 31 and Openshift Origin Build. We have deleted all the contail-container and build 31 docker images. Also deleted /etc/contrailctl. Reprovisioned the cluster using contrail-ansible with Build 32.

Workaround
-----------
Change the permission of the log file inside contrail-kube-manager docker and restarted the service

Traceback
----------
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: Traceback (most recent call last):
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/bin/contrail-kube-manager", line 9, in <module>
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: load_entry_point('kube-manager==0.1dev', 'console_scripts', 'contrail-kube-manager')()
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 158, in main
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: km_logger = logger.KubeManagerLogger(args, http_server_port=-1)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line 48, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: self.sandesh_init(http_server_port)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line 196, in sandesh_init
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: syslog_facility=self._args.syslog_facility)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_base.py", line 295, in set_logging_params
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: enable_flow_log=enable_flow_log)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 95, in set_logging_params
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: self.set_logging_file(file, maxBytes, backupCount)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 171, in set_logging_file
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: backupCount=backupCount))
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: logging.FileHandler.__init__(self, filename, mode, encoding, delay)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: StreamHandler.__init__(self, self._open())
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: stream = open(self.baseFilename, self.mode)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: IOError: [Errno 13] Permission denied: '/var/log/contrail/contrail-kube-manager.log'
Sep 25 11:58:06 5b7s18 systemd[1]: contrail-kube-manager.service: main process exited, code=exited, status=1/FAILURE

root@5b7s18(kubemanager):/# journalctl -b -u contrail-kube-manager
-- Logs begin at Mon 2017-09-25 11:57:50 PDT, end at Mon 2017-09-25 12:31:10 PDT. --
Sep 25 11:58:02 5b7s18 systemd[1]: Started Contrail kube-manager.
Sep 25 11:58:02 5b7s18 systemd[1]: Starting Contrail kube-manager...
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: 09/25/2017 11:58:06 AM [contrail-kube-manager]: SANDESH: CONNECT TO COLLECTOR: True
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: 09/25/2017 11:58:06 AM [contrail-kube-manager]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG]
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: 09/25/2017 11:58:06 AM [contrail-kube-manager]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-kube-manager.log]
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: Traceback (most recent call last):
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/bin/contrail-kube-manager", line 9, in <module>
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: load_entry_point('kube-manager==0.1dev', 'console_scripts', 'contrail-kube-manager')()
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 158, in main
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: km_logger = logger.KubeManagerLogger(args, http_server_port=-1)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line 48, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: self.sandesh_init(http_server_port)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line 196, in sandesh_init
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: syslog_facility=self._args.syslog_facility)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_base.py", line 295, in set_logging_params
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: enable_flow_log=enable_flow_log)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 95, in set_logging_params
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: self.set_logging_file(file, maxBytes, backupCount)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 171, in set_logging_file
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: backupCount=backupCount))
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: logging.FileHandler.__init__(self, filename, mode, encoding, delay)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: StreamHandler.__init__(self, self._open())
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: stream = open(self.baseFilename, self.mode)
Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: IOError: [Errno 13] Permission denied: '/var/log/contrail/contrail-kube-manager.log'
Sep 25 11:58:06 5b7s18 systemd[1]: contrail-kube-manager.service: main process exited, code=exited, status=1/FAILURE
Sep 25 11:58:06 5b7s18 kill[1109]: Usage:
Sep 25 11:58:06 5b7s18 kill[1109]: kill [options] <pid|name> [...]
Sep 25 11:58:06 5b7s18 kill[1109]: Options:
Sep 25 11:58:06 5b7s18 kill[1109]: -a, --all do not restrict the name-to-pid conversion to processes
Sep 25 11:58:06 5b7s18 kill[1109]: with the same uid as the present process
Sep 25 11:58:06 5b7s18 kill[1109]: -s, --signal <sig> send specified signal
Sep 25 11:58:06 5b7s18 kill[1109]: -q, --queue <sig> use sigqueue(2) rather than kill(2)
Sep 25 11:58:06 5b7s18 kill[1109]: -p, --pid print pids without signaling them
Sep 25 11:58:06 5b7s18 kill[1109]: -l, --list [=<signal>] list signal names, or convert one to a name
Sep 25 11:58:06 5b7s18 kill[1109]: -L, --table list signal names and numbers
Sep 25 11:58:06 5b7s18 kill[1109]: -h, --help display this help and exit
Sep 25 11:58:06 5b7s18 kill[1109]: -V, --version output version information and exit
Sep 25 11:58:06 5b7s18 kill[1109]: For more details see kill(1).
Sep 25 11:58:06 5b7s18 systemd[1]: contrail-kube-manager.service: control process exited, code=exited status=1
Sep 25 11:58:06 5b7s18 systemd[1]: Unit contrail-kube-manager.service entered failed state.
Sep 25 11:58:06 5b7s18 systemd[1]: contrail-kube-manager.service failed.
Sep 25 11:58:06 5b7s18 systemd[1]: contrail-kube-manager.service holdoff time over, scheduling restart.
Sep 25 11:58:06 5b7s18 systemd[1]: Started Contrail kube-manager.
Sep 25 11:58:06 5b7s18 systemd[1]: Starting Contrail kube-manager...

chhandak (chhandak)
Changed in juniperopenstack:
milestone: none → r4.0.1.0
assignee: nobody → Sachchidanand Vaidya (vaidyasd)
importance: Undecided → High
information type: Proprietary → Public
Sachchidanand Vaidya (vaidyasd)
Changed in juniperopenstack:
assignee: Sachchidanand Vaidya (vaidyasd) → Aniket Gawade (aniketgawade)
Revision history for this message
Sachchidanand Vaidya (vaidyasd) wrote : Re: [Bug 1719430] [NEW] Openshift : [Build 4.0.1.0-32] contrail-kube-manager failed to come up due to log permission issue
Download full text (19.7 KiB)

This is file owner issue. We mount /var/log/contrail/kubemanager/ from
host into container as /var/log/contrail/
Kube-manager container creates contrail-kube-manager.log file with
Contrail as the User.

But user ³Contrail² is local to the containers. So when we delete and
create the container again, user ³Contrail²
Will have a different userID. This is causing issue with access permission
to create log file inside container.

On 9/25/17, 1:02 PM, "<email address hidden> on behalf of chhandak"
<<email address hidden> on behalf of <email address hidden>> wrote:

>Public bug reported:
>
>While upgrading Openshift cluster from build 31 to 32 in Redhat base os,
>contrail kube-manager failed to come up for pemission issue with
>contrail-kube-manager.log
>
>What Has been tried
>---------------------
>The cluster was up with Build 31 and Openshift Origin Build. We have
>deleted all the contail-container and build 31 docker images. Also
>deleted /etc/contrailctl. Reprovisioned the cluster using
>contrail-ansible with Build 32.
>
>Workaround
>-----------
>Change the permission of the log file inside contrail-kube-manager docker
>and restarted the service
>
>
>Traceback
>----------
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: Traceback (most
>recent call last):
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/bin/contrail-kube-manager", line 9, in <module>
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>load_entry_point('kube-manager==0.1dev', 'console_scripts',
>'contrail-kube-manager')()
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line
>158, in main
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: km_logger =
>logger.KubeManagerLogger(args, http_server_port=-1)
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line
>48, in __init__
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>self.sandesh_init(http_server_port)
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/kube_manager/common/logger.py", line
>196, in sandesh_init
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>syslog_facility=self._args.syslog_facility)
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/pysandesh/sandesh_base.py", line 295,
>in set_logging_params
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>enable_flow_log=enable_flow_log)
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 95,
>in set_logging_params
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>self.set_logging_file(file, maxBytes, backupCount)
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib/python2.7/site-packages/pysandesh/sandesh_logger.py", line 171,
>in set_logging_file
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>backupCount=backupCount))
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]: File
>"/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
>Sep 25 11:58:06 5b7s18 contrail-kube-manager[1102]:
>BaseRota...

Jeba Paulaiyan (jebap)
tags: added: releasenote
Revision history for this message
Aniket Gawade (aniketgawade) wrote :

Last time I checked with chhandak it was not reproduced. Assigning back for re-producing or closing.

Revision history for this message
chhandak (chhandak) wrote :

Reproduced the issue to Aniket

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/39602
Submitter: Aniket Gawade (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/39603
Submitter: Aniket Gawade (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/39604
Submitter: Aniket Gawade (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/39604
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/0b0aa4c8227e21d2b4c31eeb094185f6aeccccf5
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 0b0aa4c8227e21d2b4c31eeb094185f6aeccccf5
Author: Aniket Gawade <email address hidden>
Date: Thu Feb 8 15:33:36 2018 -0800

Removing service run as user contrail
Making process run as contrail is creating file permission issue, so
reverting back to root

Change-Id: Ib89651a0ec8ac5aaeb35d7e46f265dfa5fe6ef42
Closes-bug: #1719430

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/39603
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/d1120851fb74723fe5c2de2b1ebf32b2764f4bd0
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit d1120851fb74723fe5c2de2b1ebf32b2764f4bd0
Author: Aniket Gawade <email address hidden>
Date: Thu Feb 8 15:33:36 2018 -0800

Removing service run as user contrail
Making process run as contrail is creating file permission issue, so
reverting back to root

Change-Id: Ib89651a0ec8ac5aaeb35d7e46f265dfa5fe6ef42
Closes-bug: #1719430

Jeba Paulaiyan (jebap)
Changed in juniperopenstack:
status: In Progress → Won't Fix
no longer affects: juniperopenstack
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.