Comment 0 for bug 1779656

Revision history for this message
Pulkit Tandon (pulkitt) wrote : [R5.0-k8s]: Network Policy which are part of "k8s-allowall" are getting deleted automatically.

R5.0-117.

5 node plain k8s+contrail setup.
3 Controller
1 Kube master
2 Compute + k8s slave

Description:
As I create new project, I see that corresponding Allow all rules get added in "k8s-allowall" network policy.
But in some time, all policies corresponding to the namespaces which I created, get deleted automatically.
Thus, this result in test case failure.

I think this might be due to the name of the namespace used.
Prior to this test, I ran some other test cases which used same name for namespaces.
In the kube manager logs, I can see following Delete request running continuously.

07/02/2018 10:36:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:36:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace non-default:e7abbdb5-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace non-default:e7abbdb5-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909

Note that the namespaces which I created are with same name "temp-ns", "new-default" and "non-default".
For current case, following is the namespace add request from the logs:
07/02/2018 10:11:10 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace temp-ns:3e4b542e-7de0-11e8-a04b-002590aaa909

I suspect that the stale Delete requests for same namespace name(but different UUID), resulted in deletion of network policies.

If that is the case, there are 2 things to inspect:
1. Why the stale namespace deletion request is running continuously?
2. Should this stale namespace deletion request result in deletion of network policies even though they have different UUID ?

For further verification, I did a restart of Kube manager.
All policies were restored after a restart as they were read fresh.
Soon after the stale DELETE, the policies were again deleted.

I have attached the kube manager and config logs.