| 2017-08-15 07:19:04 |
chhandak |
bug |
|
|
added bug |
| 2017-08-15 07:19:13 |
chhandak |
juniperopenstack: importance |
Undecided |
Critical |
|
| 2017-08-15 07:19:54 |
chhandak |
juniperopenstack: assignee |
|
Sachchidanand Vaidya (vaidyasd) |
|
| 2017-08-15 07:20:00 |
chhandak |
juniperopenstack: milestone |
|
r4.0.1.0 |
|
| 2017-08-15 07:20:20 |
chhandak |
tags |
|
contrail-kube-manager |
|
| 2017-08-15 07:20:27 |
chhandak |
tags |
contrail-kube-manager |
blocker contrail-kube-manager |
|
| 2017-08-15 07:20:47 |
chhandak |
information type |
Proprietary |
Public |
|
| 2017-08-15 07:25:34 |
chhandak |
description |
Description:
In K8S nested setup, for a service, traffic is always landing on same backend pod rather load balancing.
For Service IP, agent programmed backend PODs as the active backup instead of active-active.
Vrouter is having only one entry as next hop instead of composite next hop.
Setup Details
--------------
*Single node contrail openstack cluster. Launched 3 VM. 1 acting as K8S master and other 2 as slaves.
*Service is created with 2 backend pods, launched on each nodes (VM)
* When a 3rd POD trying to access service, it is getting redirect to one backend pod only
Debug output
-------------
2017-08-15 05:59:40,796 - WARNING - No http hit seen for one or more pods.Pls check. Hits: {'ctest-nginx-pod-80890072': 10, 'ctest-nginx-pod-36674000': 0}
> /contrail-test/common/k8s/base.py(379)validate_nginx_lb()
root@vm-test-1:~# kubectl describe svc ctest-nginx-svc-76482655 -n ctest-namespace-80473558
Name: ctest-nginx-svc-76482655
Namespace: ctest-namespace-80473558
Labels: <none>
Annotations: <none>
Selector: app=http_test
Type: ClusterIP
IP: 10.107.220.45
Port: <unset> 80/TCP
Endpoints: 10.47.255.248:80,10.47.255.249:80
Session Affinity: None
Events: <none>
root@5b8s38:~# rt --dump 1 | grep 10.107.220.45
10.107.220.45/32 32 PT - 43 -
root@5b8s38:~# nh --get 43
Id:43 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:8 Len:18
Encap Data: 02 e5 f7 0c 28 81 00 00 5e 00 01 00 81 00 00 32 08 00
root@5b8s38:~# vif --get 8
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root
vif0/8 ctest-namespace-80473558__ctest-nginx-pod-80890072 Vlan(o/i)(,S): 50/50 Bridge Index: 0 Parent:vif0/4
Type:Virtual(Vlan) HWaddr:00:00:5e:00:01:00 IPaddr:0 >>> Traffic is also coming to same POD
Vrf:1 Flags:PL3DEr QOS:-1 Ref:5
RX packets:476 bytes:26180 errors:0
TX packets:515 bytes:26532 errors:0
Drops:427 |
Description:
In K8S nested setup, for a service, traffic is always landing on same backend pod rather load balancing.
For Service IP, agent programmed backend PODs as the active backup instead of active-active.
Vrouter is having only one entry as next hop instead of composite next hop.
Setup Details
--------------
*Single node contrail openstack cluster. Launched 3 VM. 1 acting as K8S master and other 2 as slaves.
*Service is created with 2 backend pods, launched on each nodes (VM)
* When a 3rd POD trying to access service, it is getting redirect to one backend pod only
Debug output
-------------
2017-08-15 05:59:40,796 - WARNING - No http hit seen for one or more pods.Pls check. Hits: {'ctest-nginx-pod-80890072': 10, 'ctest-nginx-pod-36674000': 0}
> /contrail-test/common/k8s/base.py(379)validate_nginx_lb()
root@vm-test-1:~# kubectl describe svc ctest-nginx-svc-76482655 -n ctest-namespace-80473558
Name: ctest-nginx-svc-76482655
Namespace: ctest-namespace-80473558
Labels: <none>
Annotations: <none>
Selector: app=http_test
Type: ClusterIP
IP: 10.107.220.45
Port: <unset> 80/TCP
Endpoints: 10.47.255.248:80,10.47.255.249:80
Session Affinity: None
Events: <none>
root@5b8s38:~# rt --dump 1 | grep 10.107.220.45
10.107.220.45/32 32 PT - 43 -
root@5b8s38:~# nh --get 43
Id:43 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:8 Len:18
Encap Data: 02 e5 f7 0c 28 81 00 00 5e 00 01 00 81 00 00 32 08 00
root@5b8s38:~# vif --get 8
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root
vif0/8 ctest-namespace-80473558__ctest-nginx-pod-80890072 Vlan(o/i)(,S): 50/50 Bridge Index: 0 Parent:vif0/4
Type:Virtual(Vlan) HWaddr:00:00:5e:00:01:00 IPaddr:0 >>> Traffic is also coming to same POD
Vrf:1 Flags:PL3DEr QOS:-1 Ref:5
RX packets:476 bytes:26180 errors:0
TX packets:515 bytes:26532 errors:0
Drops:427
Agent Config
-------------
type:instance-ip name:ctest-nginx-svc-76482655__aae261e2-18df-4493-9b3f-a372c495826e instance-ip-address:10.107.220.45 instance-ip-secondary:false instance-ip-local-ip:false service-instance-ip:false service-health-check-ip:false id-perms permissions owner:cloud-admin owner-access:7 group:cloud-admin-group group-access:7 other-access:7 uuid uuid-mslong:12313511954892145811 uuid-lslong:11186839712731333230 Uuid : aae261e2-18df-4493-9b3f-a372c495826e enable:true created:2017-08-15T05:59:27 last-modified:2017-08-15T05:59:27 user-visible:true display-name:ctest-nginx-svc-76482655 Adjacencies: |
|
| 2017-08-15 07:34:45 |
chhandak |
description |
Description:
In K8S nested setup, for a service, traffic is always landing on same backend pod rather load balancing.
For Service IP, agent programmed backend PODs as the active backup instead of active-active.
Vrouter is having only one entry as next hop instead of composite next hop.
Setup Details
--------------
*Single node contrail openstack cluster. Launched 3 VM. 1 acting as K8S master and other 2 as slaves.
*Service is created with 2 backend pods, launched on each nodes (VM)
* When a 3rd POD trying to access service, it is getting redirect to one backend pod only
Debug output
-------------
2017-08-15 05:59:40,796 - WARNING - No http hit seen for one or more pods.Pls check. Hits: {'ctest-nginx-pod-80890072': 10, 'ctest-nginx-pod-36674000': 0}
> /contrail-test/common/k8s/base.py(379)validate_nginx_lb()
root@vm-test-1:~# kubectl describe svc ctest-nginx-svc-76482655 -n ctest-namespace-80473558
Name: ctest-nginx-svc-76482655
Namespace: ctest-namespace-80473558
Labels: <none>
Annotations: <none>
Selector: app=http_test
Type: ClusterIP
IP: 10.107.220.45
Port: <unset> 80/TCP
Endpoints: 10.47.255.248:80,10.47.255.249:80
Session Affinity: None
Events: <none>
root@5b8s38:~# rt --dump 1 | grep 10.107.220.45
10.107.220.45/32 32 PT - 43 -
root@5b8s38:~# nh --get 43
Id:43 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:8 Len:18
Encap Data: 02 e5 f7 0c 28 81 00 00 5e 00 01 00 81 00 00 32 08 00
root@5b8s38:~# vif --get 8
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root
vif0/8 ctest-namespace-80473558__ctest-nginx-pod-80890072 Vlan(o/i)(,S): 50/50 Bridge Index: 0 Parent:vif0/4
Type:Virtual(Vlan) HWaddr:00:00:5e:00:01:00 IPaddr:0 >>> Traffic is also coming to same POD
Vrf:1 Flags:PL3DEr QOS:-1 Ref:5
RX packets:476 bytes:26180 errors:0
TX packets:515 bytes:26532 errors:0
Drops:427
Agent Config
-------------
type:instance-ip name:ctest-nginx-svc-76482655__aae261e2-18df-4493-9b3f-a372c495826e instance-ip-address:10.107.220.45 instance-ip-secondary:false instance-ip-local-ip:false service-instance-ip:false service-health-check-ip:false id-perms permissions owner:cloud-admin owner-access:7 group:cloud-admin-group group-access:7 other-access:7 uuid uuid-mslong:12313511954892145811 uuid-lslong:11186839712731333230 Uuid : aae261e2-18df-4493-9b3f-a372c495826e enable:true created:2017-08-15T05:59:27 last-modified:2017-08-15T05:59:27 user-visible:true display-name:ctest-nginx-svc-76482655 Adjacencies: |
Description:
In K8S nested setup, for a service, traffic is always landing on same backend pod rather load balancing.
For Service IP, agent programmed backend PODs as the active backup instead of active-active.
Vrouter is having only one entry as next hop instead of composite next hop.
Setup Details
--------------
*Single node contrail openstack cluster. Launched 3 VM. 1 acting as K8S master and other 2 as slaves.
*Service is created with 2 backend pods, launched on each nodes (VM)
* When a 3rd POD trying to access service, it is getting redirect to one backend pod only
Debug output
-------------
2017-08-15 05:59:40,796 - WARNING - No http hit seen for one or more pods.Pls check. Hits: {'ctest-nginx-pod-80890072': 10, 'ctest-nginx-pod-36674000': 0}
> /contrail-test/common/k8s/base.py(379)validate_nginx_lb()
root@vm-test-1:~# kubectl describe svc ctest-nginx-svc-76482655 -n ctest-namespace-80473558
Name: ctest-nginx-svc-76482655
Namespace: ctest-namespace-80473558
Labels: <none>
Annotations: <none>
Selector: app=http_test
Type: ClusterIP
IP: 10.107.220.45
Port: <unset> 80/TCP
Endpoints: 10.47.255.248:80,10.47.255.249:80
Session Affinity: None
Events: <none>
root@5b8s38:~# rt --dump 1 | grep 10.107.220.45
10.107.220.45/32 32 PT - 43 -
root@5b8s38:~# nh --get 43
Id:43 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
Flags:Valid, Policy, Etree Root,
EncapFmly:0806 Oif:8 Len:18
Encap Data: 02 e5 f7 0c 28 81 00 00 5e 00 01 00 81 00 00 32 08 00
root@5b8s38:~# vif --get 8
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
Proxy=MAC Requests Proxied Always, Er=Etree Root
vif0/8 ctest-namespace-80473558__ctest-nginx-pod-80890072 Vlan(o/i)(,S): 50/50 Bridge Index: 0 Parent:vif0/4
Type:Virtual(Vlan) HWaddr:00:00:5e:00:01:00 IPaddr:0 >>> Traffic is also coming to same POD
Vrf:1 Flags:PL3DEr QOS:-1 Ref:5
RX packets:476 bytes:26180 errors:0
TX packets:515 bytes:26532 errors:0
Drops:427
Agent Config
-------------
type:instance-ip name:ctest-nginx-svc-76482655__aae261e2-18df-4493-9b3f-a372c495826e instance-ip-address:10.107.220.45 instance-ip-secondary:false instance-ip-local-ip:false service-instance-ip:false service-health-check-ip:false id-perms permissions owner:cloud-admin owner-access:7 group:cloud-admin-group group-access:7 other-access:7 uuid uuid-mslong:12313511954892145811 uuid-lslong:11186839712731333230 Uuid : aae261e2-18df-4493-9b3f-a372c495826e enable:true created:2017-08-15T05:59:27 last-modified:2017-08-15T05:59:27 user-visible:true display-name:ctest-nginx-svc-76482655 Adjacencies:
Api Server Loadbalancer memebers
---------------------------------
{
"loadbalancer-pool": {
"display_name": "ctest-nginx-svc-76482655__e5b2434e-817e-11e7-b94d-02a4b482907a-TCP-80-f11a51a9-5df0-40a8-ac20-6ee40b917b5b",
"fq_name": [
"default-domain",
"admin",
"ctest-nginx-svc-76482655__e5b2434e-817e-11e7-b94d-02a4b482907a-TCP-80-f11a51a9-5df0-40a8-ac20-6ee40b917b5b"
],
"href": "http://127.0.0.1:8095/loadbalancer-pool/c9b845de-c5d2-4a0d-908c-f4eb71911455",
"id_perms": {
"created": "2017-08-15T05:59:28.100687",
"creator": null,
"description": null,
"enable": true,
"last_modified": "2017-08-15T05:59:28.100687",
"permissions": {
"group": "admin",
"group_access": 7,
"other_access": 7,
"owner": "admin",
"owner_access": 7
},
"user_visible": true,
"uuid": {
"uuid_lslong": 10415969330260939861,
"uuid_mslong": 14535444620442225165
}
},
"loadbalancer_listener_refs": [
{
"attr": null,
"href": "http://127.0.0.1:8095/loadbalancer-listener/f11a51a9-5df0-40a8-ac20-6ee40b917b5b",
"to": [
"default-domain",
"admin",
"ctest-nginx-svc-76482655__e5b2434e-817e-11e7-b94d-02a4b482907a-TCP-80-f11a51a9-5df0-40a8-ac20-6ee40b917b5b"
],
"uuid": "f11a51a9-5df0-40a8-ac20-6ee40b917b5b"
}
],
"loadbalancer_members": [
{
"href": "http://127.0.0.1:8095/loadbalancer-member/8e035f17-a65a-4e36-ae03-8c91054e75b0",
"to": [
"default-domain",
"admin",
"ctest-nginx-svc-76482655__e5b2434e-817e-11e7-b94d-02a4b482907a-TCP-80-f11a51a9-5df0-40a8-ac20-6ee40b917b5b",
"8e035f17-a65a-4e36-ae03-8c91054e75b0"
],
"uuid": "8e035f17-a65a-4e36-ae03-8c91054e75b0"
},
{
"href": "http://127.0.0.1:8095/loadbalancer-member/6cc3abdd-3ca6-422d-be34-9bf7c8737471",
"to": [
"default-domain",
"admin",
"ctest-nginx-svc-76482655__e5b2434e-817e-11e7-b94d-02a4b482907a-TCP-80-f11a51a9-5df0-40a8-ac20-6ee40b917b5b",
"6cc3abdd-3ca6-422d-be34-9bf7c8737471"
],
"uuid": "6cc3abdd-3ca6-422d-be34-9bf7c8737471"
} |
|
| 2017-08-18 19:00:59 |
Sachchidanand Vaidya |
juniperopenstack: assignee |
Sachchidanand Vaidya (vaidyasd) |
Yuvaraja Mariappan (ymariappan) |
|
| 2017-08-23 02:22:23 |
Yuvaraja Mariappan |
juniperopenstack: assignee |
Yuvaraja Mariappan (ymariappan) |
Hari Prasad Killi (haripk) |
|
| 2017-08-29 09:01:01 |
OpenContrail Admin |
nominated for series |
|
juniperopenstack/r4.0 |
|
| 2017-08-29 09:01:01 |
OpenContrail Admin |
bug task added |
|
juniperopenstack/r4.0 |
|
| 2017-08-29 09:01:01 |
OpenContrail Admin |
bug task added |
|
juniperopenstack/r4.0 |
|
| 2017-08-29 11:05:28 |
Hari Prasad Killi |
juniperopenstack/r4.0: milestone |
|
r4.0.1.0 |
|
| 2017-08-29 11:05:31 |
Hari Prasad Killi |
juniperopenstack: milestone |
r4.0.1.0 |
r4.1.0.0-fcs |
|
| 2017-08-31 08:06:46 |
OpenContrail Admin |
nominated for series |
|
juniperopenstack/trunk |
|
| 2017-08-31 08:06:46 |
OpenContrail Admin |
bug task added |
|
juniperopenstack/trunk |
|
| 2017-08-31 08:06:46 |
OpenContrail Admin |
bug task added |
|
juniperopenstack/trunk |
|
| 2017-09-01 03:50:26 |
Hari Prasad Killi |
juniperopenstack/r4.0: status |
In Progress |
Fix Committed |
|
| 2017-09-01 06:01:20 |
Hari Prasad Killi |
juniperopenstack/trunk: assignee |
Hari Prasad Killi (haripk) |
Yuvaraja Mariappan (ymariappan) |
|
| 2017-09-01 14:19:17 |
Hari Prasad Killi |
juniperopenstack/trunk: status |
In Progress |
Fix Committed |
|
