[Kube Manager]: Kube manager container restarting

Bug #1777277 reported by Pulkit Tandon on 2018-06-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Critical
Ignatious Johnson Christopher
Trunk
Fix Released
Critical
Ignatious Johnson Christopher

Bug Description

R5.0-101
Contrail+K8s provisioning.

Kube manager container stuck in restart loop.

Following are the container logs:

+ mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini
+ exec /usr/bin/python /usr/bin/contrail-kube-manager -c /etc/contrail/contrail-kubernetes.conf
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: CONNECT TO COLLECTOR: True
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG]
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-kube-manager.log]
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace contrail:19fe9131-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace default:16c36eb2-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace kube-public:16c4ef25-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace kube-system:1238c513-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
PodMonitor - Got ADDED Pod kube-system:kube-dns-6f4fd4bdf-wcxvj:1be0a05e-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service default:kubernetes:16c3de06-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service kube-system:kube-dns:18a8edc5-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service kube-system:kubernetes-dashboard:45bcbbb8-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints default:kubernetes:16c469bf-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints kube-system:kube-dns:1bdcf97e-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints kube-system:kubernetes-dashboard:45bf0bef-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/usr/bin/contrail-kube-manager", line 9, in <module>
    load_entry_point('kube-manager==0.1dev', 'console_scripts', 'contrail-kube-manager')()
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 262, in main
    vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/cfgm_common/zkclient.py", line 505, in master_election
    self._election.run(func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kazoo/recipe/election.py", line 53, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 194, in run_kube_manager
    is_master=True)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 89, in __init__
    vnc_kubernetes_config_dict=vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 60, in __init__
    self.vnc_lib = self._vnc_connect()
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 170, in _vnc_connect
    auth_token_url=self.args.auth_token_url)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 508, in __init__
    _read_cfg(cfg_parser, 'global', 'curl_log', False))
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 78, in __init__
    self.log_file.basename())
AttributeError: 'str' object has no attribute 'basename'

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Review in progress for https://review.opencontrail.org/43903
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Review in progress for https://review.opencontrail.org/43937
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Review in progress for https://review.opencontrail.org/43937
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Reviewed: https://review.opencontrail.org/43902
Committed: http://github.com/Juniper/contrail-api-client/commit/6e24246a7b3de83a0c7475ab94ad952d2ea17308
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 6e24246a7b3de83a0c7475ab94ad952d2ea17308
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Jun 18 00:59:48 2018 +0000

Using the proper basename method to find the

basename of the given log file.
Also try creating the log dir if not exists.

Added UT's

Change-Id: Ib67699b17d032321535f4dac453ccc225cf90757
Closes-Bug: 1777277

Reviewed: https://review.opencontrail.org/43937
Committed: http://github.com/Juniper/contrail-api-client/commit/6c8b72993d27afa59067ec56f484e74c02c2d32f
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 6c8b72993d27afa59067ec56f484e74c02c2d32f
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Jun 18 00:59:48 2018 +0000

Using the proper basename method to find the

basename of the given log file.
Also try creating the log dir if not exists.

Added UT's

Change-Id: Ib67699b17d032321535f4dac453ccc225cf90757
Closes-Bug: 1777277

Pulkit Tandon (pulkitt) wrote :

Verified on R5.0-111.
Issue not observed. hence closing the bug

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers