Sachin,
On Sandip’s setup the basic ping test use to spawn three VM’s on three different ESXi host and few VM’s used to DHCP issue.
It always use to happen. We were not able to reproduce the issue locally even then. My understanding was the test script was able to successfully trigger the recreation.
But looks like even with test script we are not hitting the DHCP issue.
At this point test team if they can recreate the issue on Sunnyvale setup that would be helpful.

Thanks,
Ram

From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, August 18, 2018 at 5:55 PM
To: Ram Yadav <email address hidden>, Aswani Kumar Gaddam <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Ram,
   Did this test-case always use to fail ?
 Can you run this test in a loop and see if it fails.
Also Aswini has started all Sanity tests, how many have failed ? Did some of them has passed last time we ran with Centralized plugin ?

Thanks,
Sachin
From: Ram Yadav <email address hidden>
Date: Saturday, August 18, 2018 at 3:20 PM
To: Sachchidanand Vaidya <email address hidden>, Aswani Kumar Gaddam <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Looks like the basic ping between VM within a VN are passing on my setup.
This testcase used to fail on Sandip Dey setup since not all VM’s used to get IP’s.
So we are not able to re-produce the DHCP issue.
This setup is using distributed plugin from build master-221.

Sandip,
Can you look at this setup and try to recreate DHCP issue?

Thanks,
Ram

From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, August 18, 2018 at 12:09 PM
To: Aswani Kumar Gaddam <email address hidden>, Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Thanks Aswani. How did you get it to work ?

Is it the command “ bash -x run_tests.sh  -T vcenter --send-mail -U”

Thanks,
Sachin
From: Aswani Kumar Gaddam <email address hidden>
Date: Saturday, August 18, 2018 at 9:44 AM
To: Sachchidanand Vaidya <email address hidden>, Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Sachin,

I triggered the sanity in the following container and its running now

[root@a3s10 ~]# docker attach ab0e1c12eb88

[root@ab0e1c12eb88 contrail-test]# bash -x run_tests.sh  -T vcenter --send-mail -U



2018-08-18 09:41:56,829 - INFO - ================================================================================

2018-08-18 09:41:56,829 - INFO - STARTING TEST    : test_contrail_alarms

2018-08-18 09:41:56,831 - INFO - TEST DESCRIPTION :  Test to check if alarms are present





2018-08-18 09:42:00,114 - INFO - Initial checks done. Running the testcase now

2018-08-18 09:42:00,115 - INFO -

2018-08-18 09:42:03,333 - INFO - END TEST : test_contrail_alarms : PASSED[0:00:07]

2018-08-18 09:42:03,334 - INFO - --------------------------------------------------------------------------------

2018-08-18 09:42:03,339 - INFO - ================================================================================

2018-08-18 09:42:03,339 - INFO - STARTING TEST    : test_contrail_status

2018-08-18 09:42:03,339 - INFO - TEST DESCRIPTION :  Test to verify that all services are running and active



Thanks,
Aswani Kumar

From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, 18 August 2018 at 10:03 PM
To: Madhava Rao Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Ok. Single Sanity run passed.
2018-08-18 08:53:06,766 - INFO - END TEST : test_ping_within_vn : PASSED[0:26:42]
2018-08-18 08:53:06,766 - INFO - --------------------------------------------------------------------------------

But I’m trying to run complete Sanity using following command:
[root@ab0e1c12eb88 contrail-test]# ./run_tests.sh -T venter --send-mail -

The run was over immediately..
Sandip/ Sudhee, can you pls let us what the error is here ?

I have attached complete log to the mail. Relevant logs are included :

[root@10.84.16.10] sudo: systemctl enable nfs-server.service
[root@10.84.16.10] sudo: systemctl start nfs-server.service
[root@10.84.16.11] run: vim-cmd hostsvc/vmotion/vnic_set vmk0
[root@10.84.16.12] run: vim-cmd hostsvc/vmotion/vnic_set vmk0
Exception in thread Thread-8 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
[localhost] local: rpm -q --queryformat '%{VERSION}-' contrail-test; rpm -q --queryformat '%{RELEASE}' contrail-test | awk -F'.' '{print $1}'
Exception in thread Thread-3 (most likely raised during interpreter shutdown):Generated report_details* file: report_details_2018_08_18_09_29_40.ini
could not change test cases names
could not change test cases names
Buildfile: /contrail-test/build.xml

junit-report:
   [delete] Deleting directory /contrail-test/report
[junitreport] Processing /contrail-test/report/TESTS-TestSuites.xml to /contrail-test/report/junit-noframes.html
[junitreport] Loading stylesheet /contrail-test/junit-noframes.xsl
[junitreport] Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
[junitreport] Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.
[junitreport] Transform time: 324ms

BUILD SUCCESSFUL
Total time: 0 seconds
Generate HTML reports in report/ folder : report/junit-noframes.html
logs/tcutils.util.log
logs/tcutils.config.vcenter_verification.log
logs/__main__.log
logs/tcutils.collector.opserver_results.log
logs/introspect.log
logs/a3s10-tracebacks.log
logs/a3s10-ignored_tracebacks.log
logs/a3s10-traceback_parse_failures.log
logs/common.contrail_test_init.log
logs/testbasicvmvn.log
Converted log files to html files
[root@10.84.16.10] put: tools/search-bt.py -> /tmp/search-bt.py
Traceback details will be in /tmp/a3s10-tracebacks.log
Ignored Traceback details will be in /tmp/a3s10-ignored_tracebacks.log
Traceback parse errors will be in /tmp/a3s10-traceback_parse_failures.log
Done looking for tracebacks
[root@10.84.16.10] download: /contrail-test/logs/a3s10-tracebacks.log <- /tmp/a3s10-tracebacks.log

Warning: Local file /contrail-test/logs/a3s10-tracebacks.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-tracebacks.log'

Underlying exception:
    No such file

[root@10.84.16.10] download: /contrail-test/logs/a3s10-ignored_tracebacks.log <- /tmp/a3s10-ignored_tracebacks.log

Warning: Local file /contrail-test/logs/a3s10-ignored_tracebacks.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-ignored_tracebacks.log'

Underlying exception:
    No such file

[root@10.84.16.10] download: /contrail-test/logs/a3s10-traceback_parse_failures.log <- /tmp/a3s10-traceback_parse_failures.log

Warning: Local file /contrail-test/logs/a3s10-traceback_parse_failures.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-traceback_parse_failures.log'

Underlying exception:
    No such file

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Uploaded reports
Not all mail server details are available. Skipping mail send.
Sent mail to interested parties
[root@ab0e1c12eb88 contrail-test]#


Thanks,
Sachin
From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, August 18, 2018 at 8:30 AM
To: Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

After removing nfs-ds from ESXI hosts datastore, test is running now.
I will post the result once the tests are run.

Thanks
Sachin

From: Sudheendra Rao <email address hidden>
Date: Saturday, August 18, 2018 at 4:15 AM
To: Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Hi Ram,

    Sandip had already mentioned that automation code expects that vcenter_vars.yaml has ‘server1’ only under ‘vcenter_servers’, anything else will result in error.

Change all occurrence of SRV1 in vcenter_vars.yaml to server1 and launch new test container using test_runner.sh
I tried and it got past that stage, but seeing below error, I don’t understand this, please check – the test container is ab0e1c12eb88

[root@ab0e1c12eb88 contrail-test]# python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn
:
:

[root@10.84.16.10] out: Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@10.84.16.10] out:

[root@10.84.16.10] sudo: systemctl start nfs-server.service
======================================================================
ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass
    super(TestBasicVMVN, cls).setUpClass()
  File "scripts/vm_regression/base.py", line 12, in setUpClass
    super(BaseVnVmTest, cls).setUpClass()
  File "test_v1.py", line 38, in setUpClass
    cls.admin_isolated_creds.setUp()
  File "common/isolated_creds.py", line 132, in setUp
    password=self.password)
  File "common/connections.py", line 68, in __init__
    self.vnc_lib = self.get_vnc_lib_h()
  File "common/connections.py", line 167, in get_vnc_lib_h
    env[attr].setUp()
  File "/contrail-test/fixtures/vnc_api_test.py", line 113, in setUp
    if self.orch:
  File "/contrail-test/fixtures/vnc_api_test.py", line 178, in orch
    self._orch = self.get_orch_h()
  File "/contrail-test/fixtures/vnc_api_test.py", line 201, in get_orch_h
    logger=self.logger)
  File "/contrail-test/fixtures/vcenter.py", line 144, in __init__
    self._nfs_ds = NFSDatastore(self._inputs, self)
  File "tcutils/util.py", line 997, in __call__
    Singleton, cls).__call__(*args, **kwargs)
  File "/contrail-test/fixtures/vcenter.py", line 92, in __init__
    host.configManager.datastoreSystem.CreateNasDatastore(spec)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1366, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
vim.fault.DuplicateName: (vim.fault.DuplicateName) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = "The name 'nfs-ds' already exists.",
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   name = 'nfs-ds',
   object = 'vim.Datastore:10.84.16.10:/nfs'
}

Ran 0 tests in 25.285s
FAILED (failures=1)


Thanks,
Sudhee.

From: Ram Yadav <email address hidden>
Date: Saturday, 18 August 2018 at 6:21 AM
To: Sandip Dey <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Sandip Dey <email address hidden>, Marcin Orkiszewski <email address hidden>, Sudheendra Rao <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Hi Sandip,
I pulled in the docker image:
contrail-test-test:master-221

And followed the steps for single sanity test case run.
I’m getting the following error:

ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass

    super(TestBasicVMVN, cls).setUpClass()

  File "scripts/vm_regression/base.py", line 12, in setUpClass

    super(BaseVnVmTest, cls).setUpClass()

  File "test_v1.py", line 13, in setUpClass

    super(BaseTestCase_v1, cls).setUpClass()

  File "test.py", line 79, in setUpClass

    cls.inputs = ContrailTestInit(cls.input_file,logger = cls.logger)

  File "common/contrail_test_init.py", line 853, in __init__

    self.inputs = TestInputs(input_file, self.logger)

  File "tcutils/util.py", line 997, in __call__

    Singleton, cls).__call__(*args, **kwargs)

  File "common/contrail_test_init.py", line 102, in __init__

    self.parse_yml_file()

  File "common/contrail_test_init.py", line 589, in parse_yml_file

    self.vcenter_dc = _parse_vcenter.vcenter_dc

  File "common/contrail_test_init.py", line 1466, in vcenter_dc


















Test container is spawned on 10.84.16.10.

/root/contrail-ansible-deployer has the contrail_test_input.yml and vcenter_vars.yml










Thanks,

Ram









From: Ram Yadav <<email address hidden><mailto:<email address hidden>>>
Date: Tuesday, August 14, 2018 at 4:31 PM
To: Adrian Szczepański <<email address hidden><mailto:<email address hidden>>>, Amudha R <<email address hidden><mailto:<email address hidden>>>
Cc: Sachchidanand Vaidya <<email address hidden><mailto:<email address hidden>>>, Sandip Dey <<email address hidden><mailto:<email address hidden>>>, Marcin Orkiszewski <<email address hidden><mailto:<email address hidden>>>
Subject: steps to run sanity on vCenter local setup's

Hello,

Below is the instruction to run sanity

1.wget https://github.com/Juniper/contrail-test/raw/master/testrunner.sh
2.docker pull 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
3../testrunner.sh run -s -P contrail_test_input.yml,vcenter_vars.yml 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
4.cd contrail-test
5.export TEST_CONFIG_FILE=contrail_test_input_useit.yaml
6./run_tests.sh -T venter --send-mail -












To run a single test case
1.wget https://github.com/Juniper/contrail-test/raw/master/testrunner.sh
2.docker pull 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
3../testrunner.sh run -s -P contrail_test_input.yml,vcenter_vars.yml 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
4.cd contrail-test
5.export PYTHONPATH=$PATH:$PWD:$PWD/fixtures:$PWD/scripts
6.export TEST_CONFIG_FILE=contrail_test_input_useit.yaml
7.python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn

Docker pull needs to be done on ci-repo.englab.juniper.net or local repo as mentioned in the above step which is Bangalore local repo.

Apart from this you will need set of vmdk file which was copied by Sandip Dey at some sunnyvale location.

Sandip,
Can provide detail on the file’s that we need to copy say to codilime’s server locally for sanity to run on there setup?

Thanks,
Ram