[k8s-5.0.1]: In multi cluster nested environment, Pod with same name and namespace name is not reachable

[Pulkit Tandon]

R5.0.1. Build 122

Multi cluster Nested k8s provisioning.

Created Pod with name "busybox" in namespace "test" in cluster 1.
Cluster 1 Pod Subnet = 10.36.0.0/12
This Pod is reachable from any Pod within the same cluster

Created Pod with name "busybox" in namespace "test" in cluster 2.
Cluster 2 Pod Subnet = 10.52.0.0/12
This Pod is not reachable via any Pod within the cluster.

Verified that any other Pod created does not have any reachability issue until the namespace and pod names are same.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44712
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44722
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44712
Committed: http://github.com/Juniper/contrail-controller/commit/51720c39e20afd39275d8b1e83ba52f74cf535b2
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 51720c39e20afd39275d8b1e83ba52f74cf535b2
Author: Yuvaraja Mariappan <email address hidden>
Date: Tue Jul 17 12:23:26 2018 -0700

Fixed pod connectivity issue in multi-cluster

vif interfaces are created in vmi-displayname by agent.
Since pod's vmi-display name are formed using namespace
and pod name, it ended with same vif name in multi-cluster
env. Since vmi name is formed using pod name and vmi uuid,
vmi name should be used as display name.
Added code to use name as display name

Closes-bug: #1781119

Fixed pod connectivity issue in multi-cluster

Change-Id: I420d8df7ba71db6b223f125decfac6081c197147

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44722
Committed: http://github.com/Juniper/contrail-controller/commit/6ac7c681152a5917749cf12ed47d4570b153e303
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 6ac7c681152a5917749cf12ed47d4570b153e303
Author: Yuvaraja Mariappan <email address hidden>
Date: Tue Jul 17 12:23:26 2018 -0700

Fixed pod connectivity issue in multi-cluster

vif interfaces are created in vmi-displayname by agent.
Since pod's vmi-display name are formed using namespace
and pod name, it ended with same vif name in multi-cluster
env. Since vmi name is formed using pod name and vmi uuid,
vmi name should be used as display name.
Added code to use name as display name

Closes-bug: #1781119

Fixed pod connectivity issue in multi-cluster

Change-Id: I420d8df7ba71db6b223f125decfac6081c197147

[Venkatesh Velpula (vvelpula)]

Hi Sachin,Sudhee,

    You might have seen full failure sanity reports for k8s sanity. Please ignore these reports,
    Sometimes the images are properly pushed to the Bangalore repository by the time the sanity is triggered ..hence the provisioning fails and sanity

STDERR:
Error response from daemon: manifest for 10.204.217.152:5000/contrail-kubernetes-cni-init:queens-5.0-247 not found
MSG:
non-zero return code
failed: [10.204.217.100] (item=contrail-kubernetes-cni-init) => {
    "changed": true,
    "cmd": "docker pull 10.204.217.152:5000/contrail-kubernetes-cni-init:queens-5.0-247",
    "delta": "0:00:00.106903",
    "end": "2018-09-09 15:58:39.448900",
    "item": "contrail-kubernetes-cni-init",
    "rc": 1,
    "start": "2018-09-09 15:58:39.341997"
}
Thanks
-Venky

From: "<email address hidden>" <email address hidden>
Date: Monday, September 10, 2018 at 2:30 AM
To: Contrail Software <email address hidden>
Subject: [Build CentOS Linux release 7.5.1804 Core 5.1.0-272~queens] Micro-services-Multi-Interface-Kubernetes-HA-Sanity-Ansible-Virtual Report

Test Results

Designed for use with JUnit<http://www.junit.org> and Ant<http://jakarta.apache.org/ant>.

________________________________
Properties
Build : 5.1.0-272
CoreLocation : /cs-shared/test_runs/testbed-1-vm4/2018_09_09_20_42_00
cores : {'10.204.218.101': [], '10.204.218.100': [], '10.204.217.194': [], '10.204.217.197': [], '10.204.217.198': []}
LogsLocation : http://10.204.216.50/Docs/logs/5.1.0-272_2018_09_09_20_42_00_1536525810.61/logs/
Report : http://10.204.216.50/Docs/logs/5.1.0-272_2018_09_09_20_42_00_1536525810.61/junit-noframes.html
Topology :
DISTRO : CentOS Linux release 7.5.1804 Core
SKU : queens
Config Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Control Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Compute Nodes : ['testbed-1-vm2', 'testbed-1-vm3']
WebUI Node : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Analytics Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Database Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Physical Devices : [u! "'hooper', '192.168.1.11'"]
________________________________
Summary
Tests

Failures

Errors

Skipped

Success rate

Time

19

19

0

0

0.00%

49.928

Note: failures are anticipated and checked for with assertions while errors are unanticipated.

________________________________
Packages
Note: package statistics are not computed recursively, they only sum up all of its testsuites numbers.
Name

Tests

Errors

Failures

Skipped

Time(s)

19

0

19

0

49.928

________________________________
Package
Name

Tests

Errors

Failures

Skipped

Time(s)

9

0

9

0

15.950

10

0

10

0

33.978

Back to top
________________________________
TestCase
Name

Status

Type

Time(s)

LogFile

.setUpClass (serial_scripts.k8s_scripts.test_deployment.TestDeployment)

Failure

Details

_StringException: Traceback (most recent call last):
File "serial_scripts/k8s_scripts/test_deployment.py", line 13, in setUpClass
super...

[Sunil Basker (sunilbasker)]

From build 262 – 274, manual sanity runs have been completed and the same is reflected in dashboard.
Will schedule 275 today.

Thanks
Sunil

From: Venkatesh Velpula <email address hidden>
Date: Monday, 10 September 2018 at 10:12 AM
To: Sachchidanand Vaidya <email address hidden>, Sudheendra Rao <email address hidden>
Cc: Aniket Gawade <email address hidden>, Dinesh Bakiaraj <email address hidden>, Prasanna Mucharikar <email address hidden>, Pragash Vijayaragavan <email address hidden>, Venkatraman Venkatapathy <email address hidden>, Yuvaraja Mariappan <email address hidden>, Sunil Basker B <email address hidden>
Subject: Re: [Build CentOS Linux release 7.5.1804 Core 5.1.0-272~queens] Micro-services-Multi-Interface-Kubernetes-HA-Sanity-Ansible-Virtual Report

Hi Sachin,Sudhee,

    You might have seen full failure sanity reports for k8s sanity. Please ignore these reports,
    Sometimes the images are properly pushed to the Bangalore repository by the time the sanity is triggered ..hence the provisioning fails and sanity

STDERR:
Error response from daemon: manifest for 10.204.217.152:5000/contrail-kubernetes-cni-init:queens-5.0-247 not found
MSG:
non-zero return code
failed: [10.204.217.100] (item=contrail-kubernetes-cni-init) => {
    "changed": true,
    "cmd": "docker pull 10.204.217.152:5000/contrail-kubernetes-cni-init:queens-5.0-247",
    "delta": "0:00:00.106903",
    "end": "2018-09-09 15:58:39.448900",
    "item": "contrail-kubernetes-cni-init",
    "rc": 1,
    "start": "2018-09-09 15:58:39.341997"
}
Thanks
-Venky

From: "<email address hidden>" <email address hidden>
Date: Monday, September 10, 2018 at 2:30 AM
To: Contrail Software <email address hidden>
Subject: [Build CentOS Linux release 7.5.1804 Core 5.1.0-272~queens] Micro-services-Multi-Interface-Kubernetes-HA-Sanity-Ansible-Virtual Report

Test Results

Designed for use with JUnit<http://www.junit.org> and Ant<http://jakarta.apache.org/ant>.

________________________________
Properties
Build : 5.1.0-272
CoreLocation : /cs-shared/test_runs/testbed-1-vm4/2018_09_09_20_42_00
cores : {'10.204.218.101': [], '10.204.218.100': [], '10.204.217.194': [], '10.204.217.197': [], '10.204.217.198': []}
LogsLocation : http://10.204.216.50/Docs/logs/5.1.0-272_2018_09_09_20_42_00_1536525810.61/logs/
Report : http://10.204.216.50/Docs/logs/5.1.0-272_2018_09_09_20_42_00_1536525810.61/junit-noframes.html
Topology :
DISTRO : CentOS Linux release 7.5.1804 Core
SKU : queens
Config Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Control Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Compute Nodes : ['testbed-1-vm2', 'testbed-1-vm3']
WebUI Node : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Analytics Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Database Nodes : ['testbed-1-vm4', 'testbed-1-vm5', 'testbed-1-vm1']
Physical Devices : [u! "'hooper', '192.168.1.11'"]
________________________________
Summary
Tests

Failures

Errors

Skipped

Success rate

Time

19

19

0

0

0.00%

49.928

Note: failures are anticipated and checked for wit...

[Venkatesh Velpula (vvelpula)]

.........................................................................................................................................
Join online meeting <https://meet.juniper.net/vvelpula/UV1NB1F0>
Trouble Joining? Try Skype Web App <https://meet.juniper.net/vvelpula/UV1NB1F0?sl=1>

Join by Phone
[phone],, 999080017#
ILYNC (45962),, 999080017#
88888,, 999080017#
Find a local number <https://dialin.juniper.net>

Conference ID: 999080017

Forgot your dial-in PIN? <https://dialin.juniper.net> | First online meeting? <https://support.office.microsoft.com/en-us/article/join-a-skype-for-business-meeting-3862be6d-758a-4064-a016-67c0febf3cd5?ui=en-US&rs=en-US&ad=US#OS_Type=Mac>

Please consider with whom you are communicating, including non-Juniper personnel of companies federated on Skype for Business, before sharing any confidential information. Non-disclosure agreements may apply.
.........................................................................................................................................

[Venkatesh Velpula (vvelpula)]

Rescheduling the meeting , I am having bad cold and cough

Agenda for the meeting :

  * Feature Testplan Discussion
  * Discussion on Service Function Chaining – Have few queries in this area .

Will upload the ftp and send you the link in email .

Thanks
-Venky
.........................................................................................................................................
Join online meeting <https://meet.juniper.net/vvelpula/UV1NB1F0>
Trouble Joining? Try Skype Web App <https://meet.juniper.net/vvelpula/UV1NB1F0?sl=1>

Join by Phone
[phone],, 999080017#
ILYNC (45962),, 999080017#
88888,, 999080017#
Find a local number <https://dialin.juniper.net>

Conference ID: 999080017

Forgot your dial-in PIN? <https://dialin.juniper.net> | First online meeting? <https://support.office.microsoft.com/en-us/article/join-a-skype-for-business-meeting-3862be6d-758a-4064-a016-67c0febf3cd5?ui=en-US&rs=en-US&ad=US#OS_Type=Mac>

Please consider with whom you are communicating, including non-Juniper personnel of companies federated on Skype for Business, before sharing any confidential information. Non-disclosure agreements may apply.
.........................................................................................................................................

[Venkatesh Velpula (vvelpula)]

Hi,
   I am attaching the testplan for multi interface and I few items which need to be confirmed if they need to be covered . we can go through the tesplan or you can provide your inputs through email ..also should be fine

DPDK

SRIOV

IPV6

plugin modes to be covered

openshift

Thanks
-Venky

[Sachchidanand Vaidya (vaidyasd)]

Few notes:

  1. Add sample pod output with multiple ip addresses - Venkat.
  2. Check for default gateway reachability via default n/w - Venky.
  3. Add K8S Network Policy behavior with multiple interface Pods - Venkat
  4. Clarify Service behavior with multiple pods in spec - Venkat
  5. Can we checkin CNI<---->Agent file format change to R5.0.2 Sachin/Venkat/Yuvaraja/Dinesh?

Thanks
Sachin

From: <email address hidden>
When: 8:30 PM - 10:00 PM September 25, 2018
Subject: Discussion on k8s pod multi interface and Service chaining
Location: Any available location

Rescheduling the meeting , I am having bad cold and cough

Agenda for the meeting :

· Feature Testplan Discussion

· Discussion on Service Function Chaining – Have few queries in this area .

Will upload the ftp and send you the link in email .

Thanks
-Venky
.........................................................................................................................................
Join online meeting <https://meet.juniper.net/vvelpula/UV1NB1F0>
Trouble Joining? Try Skype Web App <https://meet.juniper.net/vvelpula/UV1NB1F0?sl=1>

Join by Phone
[phone],, 999080017#
ILYNC (45962),, 999080017#
88888,, 999080017#
Find a local number <https://dialin.juniper.net>

Conference ID: 999080017

Forgot your dial-in PIN? <https://dialin.juniper.net> | First online meeting? <https://support.office.microsoft.com/en-us/article/join-a-skype-for-business-meeting-3862be6d-758a-4064-a016-67c0febf3cd5?ui=en-US&rs=en-US&ad=US#OS_Type=Mac>

Please consider with whom you are communicating, including non-Juniper personnel of companies federated on Skype for Business, before sharing any confidential information. Non-disclosure agreements may apply.
.........................................................................................................................................

[Venkatesh Velpula (vvelpula)]

Hey Venkat ,
         After kube manager restart …pods doesn’t gets created .see the below sanity results
Thanks
-Venky
From: "<email address hidden>" <email address hidden>
Date: Wednesday, October 10, 2018 at 3:02 AM
To: Venkatesh Velpula <email address hidden>
Subject: [Build CentOS Linux release 7.5.1804 Core 5.0-284~] Sanity Report

Test Results

Designed for use with JUnit<http://www.junit.org> and Ant<http://jakarta.apache.org/ant>.

________________________________
Properties
Build : 5.0-284
cores : {'10.204.217.98': [], '10.204.217.71': [], '10.204.217.101': [], '10.204.217.100': [], '10.204.217.52': []}
LogsLocation : http://10.204.216.50/Docs/logs/5.0-284_2018_10_09_20_08_54/logs/
Report : http://10.204.216.50/Docs/logs/5.0-284_2018_10_09_20_08_54/junit-noframes.html
Topology :
DISTRO : CentOS Linux release 7.5.1804 Core
SKU :
Config Nodes : ['nodeg31', 'nodec58', 'nodeg12']
Control Nodes : ['nodeg31', 'nodec58', 'nodeg12']
Compute Nodes : ['nodec60', 'nodec61']
WebUI Node : ['nodeg31', 'nodec58', 'nodeg12']
Analytics Nodes : ['nodeg31', 'nodec58', 'nodeg12']
Database Nodes : ['nodeg31', 'nodec58', 'nodeg12']
Physical Devices : [u"'hooper', '77.77.1.100'"]
________________________________
Summary
Tests

Failures

Errors

Skipped

Success rate

Time

36

20

0

0

44.44%

3838.580

Note: failures are anticipated and checked for with assertions while errors are unanticipated.

________________________________
Packages
Note: package statistics are not computed recursively, they only sum up all of its testsuites numbers.
Name

Tests

Errors

Failures

Skipped

Time(s)

36

0

20

0

3838.580

________________________________
Package
Name

Tests

Errors

Failures

Skipped

Time(s)

scripts

19

0

4

0

1617.649

serial_scripts

17

0

16

0

2220.931

Back to top
________________________________
TestCase scripts
Name

Status

Type

Time(s)

LogFile

TestService.test_kube_dns_lookup[k8s_sanity]

Success

95.501

TestIngressTLS.test_ingress_tls_1[k8s_sanity]

Success

107.733

TestFabricFWD.test_deployment_with_replica_update_for_fabric_fwd[k8s_sanity]

Success

138.224

TestPod.test_pod_public_reachability_using_snat[k8s_sanity]

Failure

Details

_StringException: Traceback (most recent call last):
_StringException: Empty attachments:
pythonlogging:''
stderr
stdout

Traceback (most recent call last):
File "tcutils/wrappers.py", line 164, in wrapper
raise TestFailed("\n ".join(errmsg))
TestFailed: Test failed: <class 'vnc_api.exceptions.RefsExistError'>
Python 2.7.5: /usr/bin/python
Tue Oct 9 20:11:36 2018

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

/contrail-test/tcutils/wrappers.py in wrapper(self=<scripts.k8s_scripts.test_pod.TestPod.test_pod_p...ability_using_snat[k8s_sanity] id=0x7f6d83239710>, *args=(), **kwargs={})
76 log.info('Initial checks done. Running the testcase now')
77 log.info('')
78 result = function(self, *args, **kwargs)
79 except KeyboardInt...