Hi Ram , I have pushed ocata-5.0-168 tags to ci-repo . The ci-repo machine is does not have much free space and is cleaned up to retain only last 15 days images . So we are pushing this old tag as a one-time activity and note that the older images will be cleaned up soon . So if in future you require older tags please make sure you get connectivity to Bangalore repo fixed or get the images from ci-repo backup from Kuljinder. The following images are not available in bangalore registry and has not been pushed . Pulling repository localhost:5000/contrail-command Error: image contrail-command:ocata-5.0-168 not found Pulling repository localhost:5000/contrail-command-deployer Error: image contrail-command-deployer:ocata-5.0-168 not found 4228d18e2f29 and ocata-5.0-168 Pulling repository localhost:5000/contrail-external-dhcp Error: image contrail-external-dhcp:ocata-5.0-168 not found Pulling repository localhost:5000/contrail-external-redis Error: image contrail-external-redis:ocata-5.0-168 not found Pulling repository localhost:5000/contrail-external-tftp Error: image contrail-external-tftp:ocata-5.0-168 not found Pulling repository localhost:5000/contrail-go Error: image contrail-go:ocata-5.0-168 not found Pulling repository localhost:5000/contrail-vrouter-plugin-mellanox-init-redhat Error: image contrail-vrouter-plugin-mellanox-init-redhat:ocata-5.0-168 not found Regards, Mithun From: Ram Yadav Date: Thursday, 23 August 2018 at 9:33 AM To: Mithun Mistry , Abhay Joshi , Sudheendra Rao , Aswani Kumar Gaddam , Jeba Paulaiyan , Vinay Vithal Mahuli Cc: Sandip Dey , Sachchidanand Vaidya , Marcin Orkiszewski , Adrian Szczepański , Amudha R , Kuljinder Singh Subject: Re: steps to run sanity on vCenter local setup's Mithun/Vinay, Did you sync the build 5.0-168 to sunnyvale registry? I tried with proper IP for bangalore registry still same timeout issue. Please expedite it. Thanks, Ram From: Mithun Mistry <>> Date: Tuesday, August 21, 2018 at 10:04 PM To: Ram Yadav <>>, Abhay Joshi <>>, Sudheendra Rao <>>, Aswani Kumar Gaddam <>>, Jeba Paulaiyan <>>, Vinay Vithal Mahuli <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>>, Kuljinder Singh <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Ram , The IP of bangalore registry is 10.204.217.152 and not 10.214.217.152 which you have used in the playbook. Please try with this IP 10.204.217.152 . Also the Sunnyvale registry is down right now and Kuljinder is working to fix it. Let us know if you face any issues . Regards, Mithun From: Ram Yadav <>> Date: Wednesday, 22 August 2018 at 10:11 AM To: Abhay Joshi <>>, Sudheendra Rao <>>, Aswani Kumar Gaddam <>>, Jeba Paulaiyan <>>, Vinay Vithal Mahuli <>>, Mithun Mistry <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Vinay/Mithun, Please copy image ocata-5.0-168 from bangalore to sunnyvale registry? Thanks, Ram From: Abhay Joshi <>> Date: Tuesday, August 21, 2018 at 2:07 PM To: Ram Yadav <>>, Sudheendra Rao <>>, Aswani Kumar Gaddam <>>, Jeba Paulaiyan <>>, Vinay Vithal Mahuli <>>, Mithun Mistry <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Vinay/Mithun, Can you please help Ram? Thanks, Abhay From: Ram Yadav <>> Date: Monday, August 20, 2018 at 5:29 PM To: Sudheendra Rao <>>, Aswani Kumar Gaddam <>>, Jeba Paulaiyan <>>, Abhay Joshi <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Abhay, I tried getting the image from Bangalore registry but it timeouts. TASK [install_contrail : update image] ************************************************************************************************************************************************************* failed: [10.84.16.23] (item=contrail-node-init) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-node-init:ocata-5.0-168", "delta": "0:00:45.032686", "end": "2018-08-20 16:04:29.251094", "item": "contrail-node-init", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:03:44.218408", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.24] (item=contrail-node-init) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-node-init:ocata-5.0-168", "delta": "0:00:45.035015", "end": "2018-08-20 16:04:29.273194", "item": "contrail-node-init", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:03:44.238179", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.23] (item=contrail-nodemgr) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-nodemgr:ocata-5.0-168", "delta": "0:00:45.030576", "end": "2018-08-20 16:05:14.792332", "item": "contrail-nodemgr", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:04:29.761756", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.24] (item=contrail-nodemgr) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-nodemgr:ocata-5.0-168", "delta": "0:00:45.029793", "end": "2018-08-20 16:05:14.790406", "item": "contrail-nodemgr", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:04:29.760613", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.23] (item=contrail-vrouter-kernel-init) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-vrouter-kernel-init:ocata-5.0-168", "delta": "0:00:45.025981", "end": "2018-08-20 16:06:00.304874", "item": "contrail-vrouter-kernel-init", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:05:15.278893", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.24] (item=contrail-vrouter-kernel-init) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-vrouter-kernel-init:ocata-5.0-168", "delta": "0:00:45.029379", "end": "2018-08-20 16:06:00.338697", "item": "contrail-vrouter-kernel-init", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:05:15.309318", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.23] (item=contrail-vrouter-agent) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-vrouter-agent:ocata-5.0-168", "delta": "0:00:45.028958", "end": "2018-08-20 16:06:45.814656", "item": "contrail-vrouter-agent", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:06:00.785698", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} failed: [10.84.16.24] (item=contrail-vrouter-agent) => {"changed": true, "cmd": "docker pull 10.214.217.152:5000/contrail-vrouter-agent:ocata-5.0-168", "delta": "0:00:45.032826", "end": "2018-08-20 16:06:45.886996", "item": "contrail-vrouter-agent", "msg": "non-zero return code", "rc": 1, "start": "2018-08-20 16:06:00.854170", "stderr": "Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout", "stderr_lines": ["Error response from daemon: Get https://10.214.217.152:5000/v1/_ping: dial tcp 10.214.217.152:5000: i/o timeout"], "stdout": "", "stdout_lines": []} to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/install_contrail.retry PLAY RECAP ***************************************************************************************************************************************************************************************** 10.84.16.10 : ok=12 changed=4 unreachable=0 failed=1 10.84.16.23 : ok=18 changed=6 unreachable=0 failed=1 10.84.16.24 : ok=18 changed=6 unreachable=0 failed=1 localhost : ok=33 changed=1 unreachable=0 failed=0 Can you ask the ci team to sync up the image from ocata-5.0-168 Bangalore registry to sunnyvale ci-repo registry? Thanks, Ram From: Ram Yadav <>> Date: Monday, August 20, 2018 at 3:08 PM To: Sudheendra Rao <>>, Aswani Kumar Gaddam <>>, Jeba Paulaiyan <>>, Abhay Joshi <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Forgot to add Jeba and abhay. From: Ram Yadav <>> Date: Monday, August 20, 2018 at 3:07 PM To: Sudheendra Rao <>>, Aswani Kumar Gaddam <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's +jeba, abhay. Hi Jeba/Abhay, Where to find release ocata-5.0-168? I couldn’t find it on ci-repo.englab.juniper.net (10.84.5.81:5010). Thanks, Ram From: Sudheendra Rao <>> Date: Monday, August 20, 2018 at 1:04 PM To: Ram Yadav <>>, Aswani Kumar Gaddam <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Ram, On this setup also with R5.0 the problem was seen and both these setups (Sandip’s & Aswani’s) are using same vcenter server. So problem doesn’t seems to be with vcenter server version. Can you please update your finding with R5.0 on your setup? Thanks, Sudhee. From: Ram Yadav <>> Date: Tuesday, 21 August 2018 at 1:26 AM To: Aswani Kumar Gaddam <>>, Sudheendra Rao <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Can we try similar setup to where the problem was recreated on 5.0 ? From: Aswani Kumar Gaddam <>> Date: Monday, August 20, 2018 at 12:41 PM To: Ram Yadav <>>, Sudheendra Rao <>> Cc: Sandip Dey <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Ram, Tried mainline-ocata-235 single interface Not able to reproduce the problem Launched like 8 vms manually and through script all of them got ip. Thanks, Aswani Kumar From: Ram Yadav <>> Date: Monday, 20 August 2018 at 11:49 PM To: Madhava Rao Sudheendra Rao <>> Cc: Sandip Dey <>>, Aswani Kumar Gaddam <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Sudhee, Let us know when this fix is checked-in, lot of people will start using the test containers and we don’t want this to be the first issue to be looked at. Thanks, Ram From: Sudheendra Rao <>> Date: Saturday, August 18, 2018 at 8:26 PM To: Ram Yadav <>> Cc: Sandip Dey <>>, Aswani Kumar Gaddam <>>, Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Ram, The error is same as that we saw earlier and we can make out that it’s not fixed. Anyways we will fix this, but is there a requirement that you have to use SRV1 only and not server1? If there is no requirement then it just a matter of changing the name in your vcenter_vars.yaml and you are not blocked from testing. Thanks, Sudhee. Sent from my iPhone On 19-Aug-2018, at 3:38 AM, Ram Yadav <>> wrote: Hi Sudhee, I picked the latest master test container from master 241, I thought Sandip removed the dependency and checked in the fix. Can you have him check-in the fix? Regards, Ram From: Sudheendra Rao <>> Date: Saturday, August 18, 2018 at 4:15 AM To: Ram Yadav <>>, Sandip Dey <>>, Aswani Kumar Gaddam <>> Cc: Sachchidanand Vaidya <>>, Marcin Orkiszewski <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Ram, Sandip had already mentioned that automation code expects that vcenter_vars.yaml has ‘server1’ only under ‘vcenter_servers’, anything else will result in error. Change all occurrence of SRV1 in vcenter_vars.yaml to server1 and launch new test container using test_runner.sh I tried and it got past that stage, but seeing below error, I don’t understand this, please check – the test container is ab0e1c12eb88 [root@ab0e1c12eb88 contrail-test]# python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn : : [root@10.84.16.10] out: Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service. [root@10.84.16.10] out: [root@10.84.16.10] sudo: systemctl start nfs-server.service ====================================================================== ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN) ---------------------------------------------------------------------- Traceback (most recent call last): File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass super(TestBasicVMVN, cls).setUpClass() File "scripts/vm_regression/base.py", line 12, in setUpClass super(BaseVnVmTest, cls).setUpClass() File "test_v1.py", line 38, in setUpClass cls.admin_isolated_creds.setUp() File "common/isolated_creds.py", line 132, in setUp password=self.password) File "common/connections.py", line 68, in __init__ self.vnc_lib = self.get_vnc_lib_h() File "common/connections.py", line 167, in get_vnc_lib_h env[attr].setUp() File "/contrail-test/fixtures/vnc_api_test.py", line 113, in setUp if self.orch: File "/contrail-test/fixtures/vnc_api_test.py", line 178, in orch self._orch = self.get_orch_h() File "/contrail-test/fixtures/vnc_api_test.py", line 201, in get_orch_h logger=self.logger) File "/contrail-test/fixtures/vcenter.py", line 144, in __init__ self._nfs_ds = NFSDatastore(self._inputs, self) File "tcutils/util.py", line 997, in __call__ Singleton, cls).__call__(*args, **kwargs) File "/contrail-test/fixtures/vcenter.py", line 92, in __init__ host.configManager.datastoreSystem.CreateNasDatastore(spec) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1366, in InvokeMethod raise obj # pylint: disable-msg=E0702 vim.fault.DuplicateName: (vim.fault.DuplicateName) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = "The name 'nfs-ds' already exists.", faultCause = , faultMessage = (vmodl.LocalizableMessage) [], name = 'nfs-ds', object = 'vim.Datastore:10.84.16.10:/nfs' } Ran 0 tests in 25.285s FAILED (failures=1) Thanks, Sudhee. From: Ram Yadav <>> Date: Saturday, 18 August 2018 at 6:21 AM To: Sandip Dey <>> Cc: Sachchidanand Vaidya <>>, Sandip Dey <>>, Marcin Orkiszewski <>>, Sudheendra Rao <>>, Adrian Szczepański <>>, Amudha R <>> Subject: Re: steps to run sanity on vCenter local setup's Hi Sandip, I pulled in the docker image: contrail-test-test:master-221 And followed the steps for single sanity test case run. I’m getting the following error: ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN) ---------------------------------------------------------------------- Traceback (most recent call last): File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass super(TestBasicVMVN, cls).setUpClass() File "scripts/vm_regression/base.py", line 12, in setUpClass super(BaseVnVmTest, cls).setUpClass() File "test_v1.py", line 13, in setUpClass super(BaseTestCase_v1, cls).setUpClass() File "test.py", line 79, in setUpClass cls.inputs = ContrailTestInit(cls.input_file,logger = cls.logger) File "common/contrail_test_init.py", line 853, in __init__ self.inputs = TestInputs(input_file, self.logger) File "tcutils/util.py", line 997, in __call__ Singleton, cls).__call__(*args, **kwargs) File "common/contrail_test_init.py", line 102, in __init__ self.parse_yml_file() File "common/contrail_test_init.py", line 589, in parse_yml_file self.vcenter_dc = _parse_vcenter.vcenter_dc File "common/contrail_test_init.py", line 1466, in vcenter_dc Test container is spawned on 10.84.16.10. /root/contrail-ansible-deployer has the contrail_test_input.yml and vcenter_vars.yml Thanks, Ram From: Ram Yadav <