[R5.0-k8s]: Firewall rules which are part of "k8s-allowall" network policy are getting deleted automatically.

Bug #1779656 reported by Pulkit Tandon on 2018-07-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Critical
Dinesh Bakiaraj
Trunk
Fix Released
Critical
Dinesh Bakiaraj

Bug Description

R5.0-117.

5 node plain k8s+contrail setup.
3 Controller
1 Kube master
2 Compute + k8s slave

Description:
As I create new project, I see that corresponding Allow all rules get added in "k8s-allowall" network policy.
But in some time, all rules corresponding to the namespaces which I created, get deleted automatically.
Thus, this result in test case failure.

I think this might be due to the name of the namespace used.
Prior to this test, I ran some other test cases which used same name for namespaces.
In the kube manager logs, I can see following Delete request running continuously.

07/02/2018 10:36:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:36:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace non-default:e7abbdb5-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:37:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace non-default:e7abbdb5-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace new-default:e5e05334-7dd5-11e8-a04b-002590aaa909
07/02/2018 10:38:43 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got DELETED Namespace temp-ns:e977c2a4-7dd5-11e8-a04b-002590aaa909

Note that the namespaces which I created are with same name "temp-ns", "new-default" and "non-default".
For current case, following is the namespace add request from the logs:
07/02/2018 10:11:10 AM [contrail-kube-manager] [DEBUG]: __default__ [SYS_DEBUG]: KubeManagerDebugLog: VncNamespace - Got ADDED Namespace temp-ns:3e4b542e-7de0-11e8-a04b-002590aaa909

I suspect that the stale Delete requests for same namespace name(but different UUID), resulted in deletion of network policies.

If that is the case, there are 2 things to inspect:
1. Why the stale namespace deletion request is running continuously?
2. Should this stale namespace deletion request result in deletion of network firewall rules even though they have different UUID ?

For further verification, I did a restart of Kube manager.
All rules were restored after a restart as they were read fresh.
Soon after the stale DELETE request, the rules were again deleted automatically.

I have attached the kube manager and config logs.

Pulkit Tandon (pulkitt) wrote :
Pulkit Tandon (pulkitt) wrote :
summary: - [R5.0-k8s]: Network Policy which are part of "k8s-allowall" are getting
- deleted automatically.
+ [R5.0-k8s]: Firewall rules which are part of "k8s-allowall" network
+ policy are getting deleted automatically.
description: updated
Dinesh Bakiaraj (dineshb) wrote :

Pragash, this is a sideeffect of project deletion failure during K8s namespace delete.

Pulkit Tandon (pulkitt) wrote :

Verified recently on R5.0-133.(5.0.1 branch)
Its working and hence can be closed.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers