k8s_5.0: egress/ingress traffic is bocked by the firewall policy when dsnat is enabled

[Venkatesh Velpula]

orchestration :k8s
build :5.0.0-27.el7.centos
provisioning :contrail-ansible

setup
master/controller :nodei21
node/minion :nodei22

with DSNAT enabled , unable to reach the public network as it being dropped by the FWPolicy .
I tried with 5.0.0-34 build also,

though default rules exists to allow the traffic to/from isolated namespace

Action Services Endpoint1 Direction Endpoint2
pass any:0-65535 any > namespace=snattest-
pass any:0-65535 namespace=snattest > any

traffic flows only if I ad additional rule to allow any to any

pass any:any:any any <> any

[root@nodei22 /]# flow -l
Flow table(size 80609280, entries 629760)

Entries: Created 4237 Added 3831 Deleted 646 Changed 244Processed 4237 Used Overflow entries 0
(Created Flows/CPU: 365 88 272 149 164 132 118 142 113 74 135 131 122 118 128 132 71 304 157 127 114 79 107 80 69 169 133 93 99 84 90 78)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   114740<=>165868 8.8.8.8:111 1 (0)
                         10.204.217.134:0
(Gen: 1, K(nh):5, Action:D(Unknown), Flags:, QOS:-1, S(nh):15, Stats:0/0,
 SPort 51981, TTL 0, Sinfo 0.0.0.0)

   165868<=>114740 10.47.255.250:111 1 (5)
                         8.8.8.8:0
(Gen: 1, K(nh):41, Action:D(FwPolicy), Flags:, QOS:-1, S(nh):41, Stats:1/98,
 SPort 65225, TTL 0, Sinfo 6.0.0.0)

contrail-version
Package Version Build-ID | Repo | RPM Name
-------------------------------------- ------------------------------ -------------
contrail-kube-manager 5.0.0-27.el7.centos @contrail

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/41656
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/41657
Submitter: Naveen N (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/41656
Committed: http://github.com/Juniper/contrail-controller/commit/44b5be1ca5a6277078cb6d98279d787a128572fa
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 44b5be1ca5a6277078cb6d98279d787a128572fa
Author: Naveen N <email address hidden>
Date: Tue Apr 10 12:19:44 2018 +0530

* Pick source VN from source IP of VM in case DSNAT

Distributed SNAT follows floating-ip feature with extra property
of being able to port NAT, in case of floating-ip we were picking
source VN from FIP network which would be ip-fabric, this was not
desired for DSNAT hence picking source VN to be native VN for flow
evaluation.

Change-Id: I5bed88854c73e3b4b51b3ae6ad2088d258187e9d
Closes-bug: #1757491, #1758983

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/41657
Committed: http://github.com/Juniper/contrail-controller/commit/ef33053913a301130440f098b7795083a9ecb117
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit ef33053913a301130440f098b7795083a9ecb117
Author: Naveen N <email address hidden>
Date: Tue Apr 10 12:19:44 2018 +0530

* Pick source VN from source IP of VM in case DSNAT

Distributed SNAT follows floating-ip feature with extra property
of being able to port NAT, in case of floating-ip we were picking
source VN from FIP network which would be ip-fabric, this was not
desired for DSNAT hence picking source VN to be native VN for flow
evaluation.

Change-Id: I5bed88854c73e3b4b51b3ae6ad2088d258187e9d
Closes-bug: #1757491, #1758983

[Venkatesh Velpula (vvelpula)]

[root@nodei24 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
combined-test fabsnattest 1/1 Running 0 1h
fabric-test fabtest 1/1 Running 0 1h
kube-system etcd-nodei24 1/1 Running 0 1h
kube-system kube-apiserver-nodei24 1/1 Running 0 1h
kube-system kube-controller-manager-nodei24 1/1 Running 0 1h
kube-system kube-dns-6f4fd4bdf-sn5b9 3/3 Running 0 1h
kube-system kube-proxy-dllmp 1/1 Running 0 1h
kube-system kube-proxy-th6df 1/1 Running 0 1h
kube-system kube-scheduler-nodei24 1/1 Running 0 1h
snat-test snattest 1/1 Running 0 1h

[root@nodei24 ~]# kubectl exec fabsnattest -n combined-test ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=34 time=44.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=34 time=43.9 ms
^C
[root@nodei24 ~]# kubectl exec fabtest -n fabric-test ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=34 time=44.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=34 time=44.7 ms
^C
[root@nodei24 ~]# kubectl exec snattest -n snat-test ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=33 time=44.4 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=33 time=44.0 ms
^C

[root@nodei24 ~]# kubectl describe namespaces
Name: combined-test
Labels: app=fabsnattest
Annotations: opencontrail.org/ip_fabric_forwarding=true
              opencontrail.org/ip_fabric_snat=true
              opencontrail.org/isolation=true
Status: Active

No resource quota.

No resource limits.

Name: contrail
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"contrail","namespace":""}}

Status: Active

No resource quota.

No resource limits.

Name: default
Labels: <none>
Annotations: <none>
Status: Active

No resource quota.

No resource limits.

Name: fabric-test
Labels: app=fabtest
Annotations: opencontrail.org/ip_fabric_forwarding=true
              opencontrail.org/isolation=true
Status: Active

No resource quota.

No resource limits.

Name: kube-public
Labels: <none>
Annotations: <none>
Status: Active

No resource quota.

No resource limits.

Name: kube-system
Labels: <none>
Annotations: <none>
Status: Active

No resource quota.

No resource limits.

Name: snat-test
Labels: app=snattest
Annotations: opencontrail.org/ip_fabric_forwarding=false
              opencontrail.org/ip_fabric_snat=true
              opencontrail.org/isolation=true
Status: Active

No resource quota.

No resource limits.

[root@nodei25 ~]# contrail-status
Pod Service Original...