Python Kubernetes client raises exception when patching/listing nodes if <none>:<none> image is present

[Angie Wang]

Brief Description
-----------------
If Kubernetes node has images without tag and digest("crictl images" shows <none>:<none>), system operations(ie.system host-label-assgin) that invoke the list/patch node via python Kubernetes client will raise the exception "Invalid value for `names`, must not be `None`". See the bug report https://github.com/kubernetes-client/python/issues/895.
However, the label is created successfully in Kubernetes because of the short-term solution in https://github.com/kubernetes/kubernetes/pull/79018.

Two approaches to work around the issue,
1. implement a replacement function of python Kubernetes client V1ContainerImage.names in sysinv(easiest)
2. implement a similar workaround solution in containerd like CRIO did in https://github.com/cri-o/cri-o/pull/2455 to not report empty name of image to Kubernetes

The workaround should be removed if long-term solutions in https://github.com/kubernetes/kubernetes/pull/79018 are made.

Severity
--------
Major

Steps to Reproduce
------------------
system host-label-assign controller-0 test-label=enabled
In sysinv log, the error "Kubernetes exception in kube_get_nodes: Invalid value for `names`, must not be `None`" appears every minute.

Expected Behavior
------------------
No error of label audit in sysinv and "system host-label-assign" works without exception.

Actual Behavior
----------------
Label audit fails every minute and system host-label-assign" fails with exception.

Reproducibility
---------------
yes

System Configuration
--------------------
Any system types

Timestamp/Logs
---------
[sysadmin@controller-0 docker(keystone_admin)]$ system host-label-assign controller-0 test=enabled
Invalid value for `names`, must not be `None`
Traceback (most recent call last):

File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data
**args)

File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
result = getattr(proxyobj, method)(ctxt, **kwargs)

File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 11078, in update_kubernetes_label
self._kube.kube_patch_node(host.hostname, body)

File "/usr/lib64/python2.7/site-packages/sysinv/common/kubernetes.py", line 165, in kube_patch_node
api_response = self._get_kubernetesclient_core().patch_node(name, body)

File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 17028, in patch_node
(data) = self.patch_node_with_http_info(name, body, **kwargs)

File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 17116, in patch_node_with_http_info
collection_formats=collection_formats)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 163, in __call_api
return_data = self.deserialize(response_data, response_type)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 236, in deserialize
return self.__deserialize(data, response_type)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 276, in __deserialize
return self.__deserialize_model(data, klass)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 620, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 276, in __deserialize
return self.__deserialize_model(data, klass)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 620, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 254, in __deserialize
for sub_data in data]

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 276, in __deserialize
return self.__deserialize_model(data, klass)

File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 622, in __deserialize_model
instance = klass(**kwargs)

File "/usr/lib/python2.7/site-packages/kubernetes/client/models/v1_container_image.py", line 52, in _init_
self.names = names

File "/usr/lib/python2.7/site-packages/kubernetes/client/models/v1_container_image.py", line 77, in names
raise ValueError("Invalid value for `names`, must not be `None`")

Test Activity
-------------
Testing

[Ghada Khalil (gkhalil)]

stx.5.0 / medium - workaround is to remove the none/none image

[Ghada Khalil (gkhalil)]

Fixed by: https://review.opendev.org/c/starlingx/config/+/764098
Merged on 2020-11-27

[Ghada Khalil (gkhalil)]

The equivalent change is required in the nfv vim component which also uses the k8s client

[Angie Wang (angiewang)]

The change in nfv component is fixed by: https://review.opendev.org/c/starlingx/nfv/+/765036

Merged on 2020-12-02