Juniper Openstack

k8s_5.0: contrail-kube-manager container fail to start

Bug #1756344 reported by Venkatesh Velpula on 2018-03-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Status tracked in Trunk
	Trunk	Fix Released	Critical	Édouard Thuleau	Juniper Openstack r5.0.0

Bug Description

Orchestrator :8ks
build version: 5.0.0-30.el7.centos

[root@nodei23 ~]# kubectl get nodes -n kube-system
NAME STATUS ROLES AGE VERSION
nodei23 NotReady master 59m v1.9.2
nodei24 Ready <none> 59m v1.9.2
nodei25 Ready <none> 58m v1.9.2
[root@nodei23 ~]#
[root@nodei23 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
contrail-kube-manager-4qx9s 0/1 CrashLoopBackOff 16 59m
etcd-nodei23 1/1 Running 0 54m
kube-apiserver-nodei23 1/1 Running 0 54m
kube-controller-manager-nodei23 1/1 Running 0 54m
kube-dns-6f4fd4bdf-9ntbj 0/3 ContainerCreating 0 59m
kube-proxy-8fts6 1/1 Running 0 59m
kube-proxy-w7rvc 1/1 Running 0 59m
kube-proxy-wpvrj 1/1 Running 0 59m
kube-scheduler-nodei23 1/1 Running 0 54m

kubemanager logs
================
kubectl logs contrail-kube-manager-4qx9s -n kube-system
INFO Fri Mar 16 13:07:16 UTC 2018: waiting for API servers: 10.204.217.135
INFO Fri Mar 16 13:07:16 UTC 2018: waiting for API server 10.204.217.135: 0 / 120
INFO Fri Mar 16 13:07:19 UTC 2018: API server 10.204.217.135 is ready.
03/16/2018 01:07:19 PM [contrail-kube-manager]: SANDESH: CONNECT TO COLLECTOR: True
03/16/2018 01:07:19 PM [contrail-kube-manager]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE]
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html

NamespaceMonitor - Got ADDED Namespace default:766878c3-2918-11e8-83fd-002590e781be
NamespaceMonitor - Got ADDED Namespace kube-public:78e93904-2918-11e8-83fd-002590e781be
NamespaceMonitor - Got ADDED Namespace kube-system:76fa0c1b-2918-11e8-83fd-002590e781be
PodMonitor - Got ADDED Pod kube-system:kube-dns-6f4fd4bdf-9ntbj:7df76324-2918-11e8-83fd-002590e781be
ServiceMonitor - Got ADDED Service default:kubernetes:78d32024-2918-11e8-83fd-002590e781be
ServiceMonitor - Got ADDED Service kube-system:kube-dns:7ad10492-2918-11e8-83fd-002590e781be
EndPointMonitor - Got ADDED Endpoints default:kubernetes:78deac27-2918-11e8-83fd-002590e781be
EndPointMonitor - Got ADDED Endpoints kube-system:kube-dns:7e0bde08-2918-11e8-83fd-002590e781be
Traceback (most recent call last):
  File "/usr/bin/contrail-kube-manager", line 9, in <module>
    load_entry_point('kube-manager==0.1dev', 'console_scripts', 'contrail-kube-manager')()
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 262, in main
    vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/cfgm_common/zkclient.py", line 504, in master_election
    self._election.run(func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kazoo/recipe/election.py", line 53, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 194, in run_kube_manager
    is_master=True)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 89, in __init__
    vnc_kubernetes_config_dict=vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 139, in __init__
    VncSecurityPolicy.create_deny_all_security_policy()
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_security_policy.py", line 1015, in create_deny_all_security_policy
    None, None, tag_last=True, is_global=True)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_security_policy.py", line 598, in create_firewall_policy
    cls.vnc_lib.firewall_policy_update(fw_policy_obj)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 48, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 652, in _object_update
    data=prop_coll_json)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 925, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 1036, in _request
    raise HttpError(status, content)
vnc_api.exceptions.HttpError: HTTP Status: 500 Content: Internal Server Error

contrail-api logs
================

03/16/2018 12:47:52 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = nodei23:Config:contrail-api:0 client_info = << status = Idle successful_connections = 4 pid = 1 http_port = 8084 start_time = 1521204184529185 collector_name = collector_ip = 10.204.217.135:8086 collector_list = [ 10.204.217.135:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >>
03/16/2018 12:47:56 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodei23 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.204.217.135:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.204.217.135:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.204.217.135:9161, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.204.217.135:5672, ] status = Up description = >>, ] description = Collector connection down >>, ] >>
03/16/2018 12:50:10 PM [contrail-api]: __default__ [SYS_ERR]: VncApiError: <type 'exceptions.AttributeError'>
Python 2.7.5: /usr/bin/python
Fri Mar 16 12:50:10 2018

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={})
1940 (code, err_msg) = status
1941 raise cfgm_common.exceptions.HttpError(code, err_msg)
1942 response = handler(*args, **kwargs)
1943 self._generate_rest_api_response_trace(trace, response)
1944
response undefined
handler = <bound method VncApiServer.prop_collection_http_...i_server.vnc_cfg_api_server.VncApiServer object>>
args = ()
kwargs = {}

/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in prop_collection_http_post(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>)
2444 self._put_common('prop-collection-update', obj_type, obj_uuid,
2445 db_obj_dict,
2446 req_prop_coll_updates=request_params.get('updates'))
2447 # end prop_collection_http_post
2448
req_prop_coll_updates undefined
request_params = {'updates': [{'field': 'annotations', 'operation': 'set', 'position': 'project', 'value': {'key': 'project', 'value': 'default'}}, {'field': 'annotations', 'operation': 'set', 'position': 'cluster', 'value': {'key': 'cluster', 'value': 'k8s'}}, {'field': 'annotations', 'operation': 'set', 'position': 'kind', 'value': {'key': 'kind', 'value': 'VncSecurityPolicy'}}, {'field': 'annotations', 'operation': 'set', 'position': 'name', 'value': {'key': 'name', 'value': 'k8s-denyall'}}, {'field': 'annotations', 'operation': 'set', 'position': 'owner', 'value': {'key': 'owner', 'value': 'k8s'}}, {'field': 'annotations', 'operation': 'set', 'position': 'namespace', 'value': {'key': 'namespace', 'value': None}}, {'field': 'annotations', 'operation': 'set', 'position': 'tail', 'value': {'key': 'tail', 'value': 'True'}}, {'field': 'annotations', 'operation': 'set', 'position': 'spec', 'value': {'key': 'spec', 'value': 'null'}}], 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}
request_params.get = <built-in method get of dict object>

/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in _put_common(self=<vnc_cfg_api_server.vnc_cfg_api_server.VncApiServer object>, api_name='prop-collection-update', obj_type='firewall_policy', obj_uuid='da9bba2a-32da-43da-923c-876854cab754', db_obj_dict={'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2018-03-16T12:50:10.798889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2018-03-16T12:50:10.798889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10537446110309037908L, 'uuid_mslong': 15752388812062999514L}}, 'parent_type': 'policy-management', 'parent_uuid': '6f9df98b-b293-4dfa-9392-d8b8a10df6ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}, req_obj_dict=None, req_prop_coll_updates=[{'field': 'annotations', 'operation': 'set', 'position': 'project', 'value': {'key': 'project', 'value': 'default'}}, {'field': 'annotations', 'operation': 'set', 'position': 'cluster', 'value': {'key': 'cluster', 'value': 'k8s'}}, {'field': 'annotations', 'operation': 'set', 'position': 'kind', 'value': {'key': 'kind', 'value': 'VncSecurityPolicy'}}, {'field': 'annotations', 'operation': 'set', 'position': 'name', 'value': {'key': 'name', 'value': 'k8s-denyall'}}, {'field': 'annotations', 'operation': 'set', 'position': 'owner', 'value': {'key': 'owner', 'value': 'k8s'}}, {'field': 'annotations', 'operation': 'set', 'position': 'namespace', 'value': {'key': 'namespace', 'value': None}}, {'field': 'annotations', 'operation': 'set', 'position': 'tail', 'value': {'key': 'tail', 'value': 'True'}}, {'field': 'annotations', 'operation': 'set', 'position': 'spec', 'value': {'key': 'spec', 'value': 'null'}}], ref_args=None, quota_dict=None)
3846 # Permit abort resource deletion and retrun 202 status code
3847 get_context().set_state('PENDING_DBE_UPDATE')
3848 ok, result = r_class.pending_dbe_update(db_obj_dict, req_obj_dict)
3849 if not ok:
3850 code, msg = result
ok undefined
result undefined
r_class = <class 'vnc_cfg_api_server.vnc_cfg_types.FirewallPolicyServer'>
r_class.pending_dbe_update = <bound method __metaclass__.pending_dbe_update o..._api_server.vnc_cfg_types.FirewallPolicyServer'>>
db_obj_dict = {'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2018-03-16T12:50:10.798889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2018-03-16T12:50:10.798889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10537446110309037908L, 'uuid_mslong': 15752388812062999514L}}, 'parent_type': 'policy-management', 'parent_uuid': '6f9df98b-b293-4dfa-9392-d8b8a10df6ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}
req_obj_dict = None

/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_types.py in pending_dbe_update(cls=<class 'vnc_cfg_api_server.vnc_cfg_types.FirewallPolicyServer'>, obj_dict={'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2018-03-16T12:50:10.798889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2018-03-16T12:50:10.798889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10537446110309037908L, 'uuid_mslong': 15752388812062999514L}}, 'parent_type': 'policy-management', 'parent_uuid': '6f9df98b-b293-4dfa-9392-d8b8a10df6ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}, delta_obj_dict=None)
  359 @classmethod
  360 def pending_dbe_update(cls, obj_dict, delta_obj_dict={}):
  361 return cls._pending_update(obj_dict, delta_obj_dict)
  362
  363 @classmethod
cls = <class 'vnc_cfg_api_server.vnc_cfg_types.FirewallPolicyServer'>
cls._pending_update = <bound method __metaclass__._pending_update of <..._api_server.vnc_cfg_types.FirewallPolicyServer'>>
obj_dict = {'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2018-03-16T12:50:10.798889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2018-03-16T12:50:10.798889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10537446110309037908L, 'uuid_mslong': 15752388812062999514L}}, 'parent_type': 'policy-management', 'parent_uuid': '6f9df98b-b293-4dfa-9392-d8b8a10df6ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}
delta_obj_dict = None

/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_types.py in _pending_update(cls=<class 'vnc_cfg_api_server.vnc_cfg_types.FirewallPolicyServer'>, obj_dict={'display_name': 'k8s-denyall', 'fq_name': ['default-policy-management', 'k8s-denyall'], 'id_perms': {'created': '2018-03-16T12:50:10.798889', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2018-03-16T12:50:10.798889', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10537446110309037908L, 'uuid_mslong': 15752388812062999514L}}, 'parent_type': 'policy-management', 'parent_uuid': '6f9df98b-b293-4dfa-9392-d8b8a10df6ed', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'da9bba2a-32da-43da-923c-876854cab754'}, delta_obj_dict=None)
  371 return True, ''
  372
  373 delta_obj_dict.pop('uuid', None)
  374
  375 ok, result = cls._get_dedicated_draft_policy_management(obj_dict)
delta_obj_dict = None
delta_obj_dict.pop undefined
builtinNone = None
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'pop'
    __class__ = <type 'exceptions.AttributeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.AttributeError object>
    __dict__ = {}
    __doc__ = 'Attribute not found.'
    __format__ = <built-in method __format__ of exceptions.AttributeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.AttributeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.AttributeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.AttributeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.AttributeError object>
    __init__ = <method-wrapper '__init__' of exceptions.AttributeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.AttributeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.AttributeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.AttributeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.AttributeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.AttributeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.AttributeError object>
    __str__ = <method-wrapper '__str__' of exceptions.AttributeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.AttributeError object>
    args = ("'NoneType' object has no attribute 'pop'",)
    message = "'NoneType' object has no attribute 'pop'"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1942, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 2446, in prop_collection_http_post
    req_prop_coll_updates=request_params.get('updates'))
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 3848, in _put_common
    ok, result = r_class.pending_dbe_update(db_obj_dict, req_obj_dict)
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_types.py", line 361, in pending_dbe_update
    return cls._pending_update(obj_dict, delta_obj_dict)
  File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_types.py", line 373, in _pending_update
    delta_obj_dict.pop('uuid', None)
AttributeError: 'NoneType' object has no attribute 'pop'

Tags:

Sachin Bansal (sbansal) on 2018-03-16

Changed in juniperopenstack:
assignee:	nobody → Sachchidanand Vaidya (vaidyasd)

Sachchidanand Vaidya (vaidyasd) on 2018-03-17

Changed in juniperopenstack:
assignee:	Sachchidanand Vaidya (vaidyasd) → Dinesh Bakiaraj (dineshb)

Dinesh Bakiaraj (dineshb) on 2018-03-17

Changed in juniperopenstack:
assignee:	Dinesh Bakiaraj (dineshb) → Édouard Thuleau (ethuleau)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-03-19: [Review update] master

Review in progress for https://review.opencontrail.org/40800
Submitter: Édouard Thuleau (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2018-03-19: A change has been merged

Reviewed: https://review.opencontrail.org/40800
Committed: http://github.com/Juniper/contrail-controller/commit/30519c41a064331c48e6f47aaff8ceed67aaaf9d
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 30519c41a064331c48e6f47aaff8ceed67aaaf9d
Author: Édouard Thuleau <email address hidden>
Date: Mon Mar 19 09:09:25 2018 +0100

[config] Fix property collection update for draft security resources

Allow to modify resource property collection even if draft security mode
is enable.

Change-Id: I9cad321d94a1d142b47bbf85346743df23e76870
Closes-Bug: #1756344

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.