[k8s-R5.0]: Seeing VNC API errors in contrail-api and contrail-kube-manager

Bug #1764508 reported by Pulkit Tandon on 2018-04-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Medium
Pragash Vijayaragavan
Trunk
Fix Released
Medium
Pragash Vijayaragavan

Bug Description

Configuration:
K8s 1.9.2
coat-5.0-15
Centos-7.4

Setup:
5 node setup.
1 Kube master. 3 Controller.
2 Agent+ K8s slaves

The issues was observed in a k8s sanity run:
LogsLocation : http://10.204.216.50/Docs/logs/5.0-15_2018_04_16_17_17_20_1523889114.59/logs/
Report : http://10.204.216.50/Docs/logs/5.0-15_2018_04_16_17_17_20_1523889114.59/junit-noframes.html

Observed VncApi error print in contrail-api:
```
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9b3a2c84-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9ad3a586-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9b0f3858-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (2396b072-417f-11e8-854b-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
```

Also observed related errors on contrail-kube-manager:
```
 /usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py in _update_service_public_ip(self=<kube_manager.vnc.vnc_service.VncService object>, service_id=u'aa695ffc-4182-11e8-97b2-002590c55f6a', service_name=u'ctest-nginx-svc-55455654', service_namespace=u'ctest-namespace-32304451', service_type=u'LoadBalancer', external_ips=[u'10.204.221.163'], loadBalancerIp=None)
  363
  364 if service_type in ["LoadBalancer"]:
  365 if len(allocated_fips) is 0:
  366 # Allocate floating-ip from public-pool, if none exists.
  367 # if "loadBalancerIp" if specified in Service definition, allocate
builtinlen = <built-in function len>
allocated_fips = None
<type 'exceptions.TypeError'>: object of type 'NoneType' has no len()
    __class__ = <type 'exceptions.TypeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.TypeError object>
    __dict__ = {}
    __doc__ = 'Inappropriate argument type.'
    __format__ = <built-in method __format__ of exceptions.TypeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.TypeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.TypeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.TypeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.TypeError object>
    __init__ = <method-wrapper '__init__' of exceptions.TypeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.TypeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.TypeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.TypeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.TypeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.TypeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.TypeError object>
    __str__ = <method-wrapper '__str__' of exceptions.TypeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.TypeError object>
    args = ("object of type 'NoneType' has no len()",)
    message = "object of type 'NoneType' has no len()"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 525, in vnc_process
    self.service_mgr.process(event)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 588, in process
    service_type, externalIps, loadBalancerIp)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 449, in vnc_service_add
    service_namespace, service_type, externalIps, loadBalancerIp)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 365, in _update_service_public_ip
    if len(allocated_fips) is 0:
TypeError: object of type 'NoneType' has no len()
```

Pulkit Tandon (pulkitt) wrote :

Reducing the importance as it is a sporadic issue

Sachchidanand Vaidya (vaidyasd) wrote :

Can you add defensive error check for this bug.

Sachchidanand Vaidya (vaidyasd) wrote :

Pls add defensive check for these bugs.

This issue is due to the function _read_allocated_floating_ips returning None in certain cases, should update _update_service_public_ip and _read_allocated_floating_ips

we might be able to hit this issue if service is frequently deleted and added so that its ip is allocated from floating ip pool which can be exhausted

will commit code for review once my Gerrit issue is resolved

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

Review in progress for https://review.opencontrail.org/43169
Submitter: Pragash Vijayaragavan (<email address hidden>)

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

will look into master UT failure after this change is merged : https://review.opencontrail.org/#/c/43254/,

Reviewed: https://review.opencontrail.org/43169
Committed: http://github.com/Juniper/contrail-controller/commit/7b244ece4191b4b555fbd08b64df961716fdb85b
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 7b244ece4191b4b555fbd08b64df961716fdb85b
Author: Pragash Vijayaragavan <email address hidden>
Date: Tue May 15 14:55:30 2018 -0700

Added notices in fip pool get

Added notices if user specified and public fip pool is not present.

Added defensive checks to vncapi errors seen in contrail-api
and contrail-kube-manager.

Change-Id: I0b14f0c3b0e116ee350760049c391b09210ed0d7
Partial-Bug: #1764508

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

Reviewed: https://review.opencontrail.org/43142
Committed: http://github.com/Juniper/contrail-controller/commit/680c3883c9f05611f71e314bb87d38cd15924c31
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 680c3883c9f05611f71e314bb87d38cd15924c31
Author: Pragash Vijayaragavan <email address hidden>
Date: Tue May 15 14:55:30 2018 -0700

Modified notices as per comments

removed tabs

Change-Id: I0b14f0c3b0e116ee350760049c391b09210ed0d7
Partial-Bug: #1764508

Pulkit Tandon (pulkitt) wrote :

Its a non reproducible bug which was observed in couple of old builds.
Defensive checks/notices have already been included as part of fix.

Closing this bug until will see further occurrence.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers