[k8s-R5.0]: Seeing VNC API errors in contrail-api and contrail-kube-manager

[Pulkit Tandon]

Configuration:
K8s 1.9.2
coat-5.0-15
Centos-7.4

Setup:
5 node setup.
1 Kube master. 3 Controller.
2 Agent+ K8s slaves

The issues was observed in a k8s sanity run:
LogsLocation : http://10.204.216.50/Docs/logs/5.0-15_2018_04_16_17_17_20_1523889114.59/logs/
Report : http://10.204.216.50/Docs/logs/5.0-15_2018_04_16_17_17_20_1523889114.59/junit-noframes.html

Observed VncApi error print in contrail-api:
```
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9b3a2c84-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9ad3a586-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (9b0f3858-418a-11e8-b61e-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
04/16/2018 04:04:30 PM [contrail-api]: __default__ [SYS_WARN]: VncApiError: Multiget result contains a key (2396b072-417f-11e8-854b-002590c55f6a) with an empty value. obj_uuid_table: keys ([u'2327b6c2-417f-11e8-854b-002590c55f6a', u'23622f3c-417f-11e8-854b-002590c55f6a', u'2396b072-417f-11e8-854b-002590c55f6a', u'9ad3a586-418a-11e8-b61e-002590c55f6a', u'9b0f3858-418a-11e8-b61e-002590c55f6a', u'9b3a2c84-418a-11e8-b61e-002590c55f6a']), columns (None), start (backref:floating_ip:), finish (backref:floating_ip;)
```

Also observed related errors on contrail-kube-manager:
```
 /usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py in _update_service_public_ip(self=<kube_manager.vnc.vnc_service.VncService object>, service_id=u'aa695ffc-4182-11e8-97b2-002590c55f6a', service_name=u'ctest-nginx-svc-55455654', service_namespace=u'ctest-namespace-32304451', service_type=u'LoadBalancer', external_ips=[u'10.204.221.163'], loadBalancerIp=None)
  363
  364 if service_type in ["LoadBalancer"]:
  365 if len(allocated_fips) is 0:
  366 # Allocate floating-ip from public-pool, if none exists.
  367 # if "loadBalancerIp" if specified in Service definition, allocate
builtinlen = <built-in function len>
allocated_fips = None
<type 'exceptions.TypeError'>: object of type 'NoneType' has no len()
    __class__ = <type 'exceptions.TypeError'>
    __delattr__ = <method-wrapper '__delattr__' of exceptions.TypeError object>
    __dict__ = {}
    __doc__ = 'Inappropriate argument type.'
    __format__ = <built-in method __format__ of exceptions.TypeError object>
    __getattribute__ = <method-wrapper '__getattribute__' of exceptions.TypeError object>
    __getitem__ = <method-wrapper '__getitem__' of exceptions.TypeError object>
    __getslice__ = <method-wrapper '__getslice__' of exceptions.TypeError object>
    __hash__ = <method-wrapper '__hash__' of exceptions.TypeError object>
    __init__ = <method-wrapper '__init__' of exceptions.TypeError object>
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of exceptions.TypeError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of exceptions.TypeError object>
    __repr__ = <method-wrapper '__repr__' of exceptions.TypeError object>
    __setattr__ = <method-wrapper '__setattr__' of exceptions.TypeError object>
    __setstate__ = <built-in method __setstate__ of exceptions.TypeError object>
    __sizeof__ = <built-in method __sizeof__ of exceptions.TypeError object>
    __str__ = <method-wrapper '__str__' of exceptions.TypeError object>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of exceptions.TypeError object>
    args = ("object of type 'NoneType' has no len()",)
    message = "object of type 'NoneType' has no len()"

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 525, in vnc_process
    self.service_mgr.process(event)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 588, in process
    service_type, externalIps, loadBalancerIp)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 449, in vnc_service_add
    service_namespace, service_type, externalIps, loadBalancerIp)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_service.py", line 365, in _update_service_public_ip
    if len(allocated_fips) is 0:
TypeError: object of type 'NoneType' has no len()
```

[Pulkit Tandon (pulkitt)]

Reducing the importance as it is a sporadic issue

[Sachchidanand Vaidya (vaidyasd)]

Can you add defensive error check for this bug.

[Sachchidanand Vaidya (vaidyasd)]

Pls add defensive check for these bugs.

[Pragash Vijayaragavan (pvijayaragav)]

This issue is due to the function _read_allocated_floating_ips returning None in certain cases, should update _update_service_public_ip and _read_allocated_floating_ips

[Pragash Vijayaragavan (pvijayaragav)]

we might be able to hit this issue if service is frequently deleted and added so that its ip is allocated from floating ip pool which can be exhausted

[Pragash Vijayaragavan (pvijayaragav)]

will commit code for review once my Gerrit issue is resolved

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43169
Submitter: Pragash Vijayaragavan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

[Pragash Vijayaragavan (pvijayaragav)]

will look into master UT failure after this change is merged : https://review.opencontrail.org/#/c/43254/,

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/43169
Committed: http://github.com/Juniper/contrail-controller/commit/7b244ece4191b4b555fbd08b64df961716fdb85b
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 7b244ece4191b4b555fbd08b64df961716fdb85b
Author: Pragash Vijayaragavan <email address hidden>
Date: Tue May 15 14:55:30 2018 -0700

Added notices in fip pool get

Added notices if user specified and public fip pool is not present.

Added defensive checks to vncapi errors seen in contrail-api
and contrail-kube-manager.

Change-Id: I0b14f0c3b0e116ee350760049c391b09210ed0d7
Partial-Bug: #1764508

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43142
Submitter: Pragash Vijayaragavan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/43142
Committed: http://github.com/Juniper/contrail-controller/commit/680c3883c9f05611f71e314bb87d38cd15924c31
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 680c3883c9f05611f71e314bb87d38cd15924c31
Author: Pragash Vijayaragavan <email address hidden>
Date: Tue May 15 14:55:30 2018 -0700

Modified notices as per comments

removed tabs

Change-Id: I0b14f0c3b0e116ee350760049c391b09210ed0d7
Partial-Bug: #1764508

[Pulkit Tandon (pulkitt)]

Its a non reproducible bug which was observed in couple of old builds.
Defensive checks/notices have already been included as part of fix.

Closing this bug until will see further occurrence.