I rebooted both contrailVM’s to get the IP back. But still the question remain what caused it.
Only thing run before getting into this state was sanity.

From: Ram Yadav <email address hidden>
Date: Sunday, August 19, 2018 at 10:19 PM
To: Aswani Kumar Gaddam <email address hidden>, Sachchidanand Vaidya <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Aswani,
Looks like the IP address for contrail VM got changed messed up.
Does the sanity run play around with contrailVM IP’s?
IP address got unassigned for both VM’s.

Thanks,
Ram

From: Ram Yadav <email address hidden>
Date: Sunday, August 19, 2018 at 10:08 PM
To: Aswani Kumar Gaddam <email address hidden>, Sachchidanand Vaidya <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Aswani,
In the same container I’m running single test case and it fails:
[root@a3s10 ~]# docker attach ab0e1c12eb88
[root@ab0e1c12eb88 contrail-test]# export PYTHONPATH=$PATH:$PWD:$PWD/fixtures:$PWD/scripts
[root@ab0e1c12eb88 contrail-test]# export TEST_CONFIG_FILE=contrail_test_input.yaml
[root@ab0e1c12eb88 contrail-test]# python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn
Tests running...
======================================================================
ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass
    super(TestBasicVMVN, cls).setUpClass()
  File "scripts/vm_regression/base.py", line 12, in setUpClass
    super(BaseVnVmTest, cls).setUpClass()
  File "test_v1.py", line 13, in setUpClass
    super(BaseTestCase_v1, cls).setUpClass()
  File "test.py", line 79, in setUpClass
    cls.inputs = ContrailTestInit(cls.input_file,logger = cls.logger)
  File "common/contrail_test_init.py", line 853, in __init__
    self.inputs = TestInputs(input_file, self.logger)
  File "tcutils/util.py", line 997, in __call__
    Singleton, cls).__call__(*args, **kwargs)
  File "common/contrail_test_init.py", line 102, in __init__
    self.parse_yml_file()
  File "common/contrail_test_init.py", line 450, in parse_yml_file
    self.parse_topo()
  File "common/contrail_test_init.py", line 302, in parse_topo
    hostname = self.run_cmd_on_server(host_data['host_ip'], 'hostname')
  File "common/contrail_test_init.py", line 834, in run_cmd_on_server
    shell_prefix=shell_prefix)
  File "tcutils/util.py", line 814, in run_cmd_on_server
    output = _run(updated_cmd, pty=pty)
  File "/usr/lib/python2.7/site-packages/fabric/network.py", line 633, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fabric/operations.py", line 1095, in sudo
    stderr=stderr, timeout=timeout, shell_escape=shell_escape,
  File "/usr/lib/python2.7/site-packages/fabric/operations.py", line 909, in _run_command
    channel=default_channel(), command=wrapped_command, pty=pty,
  File "/usr/lib/python2.7/site-packages/fabric/state.py", line 387, in default_channel
    chan = _open_session()
  File "/usr/lib/python2.7/site-packages/fabric/state.py", line 379, in _open_session
    return connections[env.host_string].get_transport().open_session()
  File "/usr/lib/python2.7/site-packages/fabric/network.py", line 154, in __getitem__
    self.connect(key)
  File "/usr/lib/python2.7/site-packages/fabric/network.py", line 146, in connect
    user, host, port, cache=self, seek_gateway=seek_gateway)
  File "/usr/lib/python2.7/site-packages/fabric/network.py", line 559, in connect
    raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to 10.84.16.23 (tried 1 time)

Ran 0 tests in 10.044s
FAILED (failures=1)



What could be the issue.
But similar ping test case passed in the sanity run.

Thanks,
Ram

From: Aswani Kumar Gaddam <email address hidden>
Date: Saturday, August 18, 2018 at 9:44 AM
To: Sachchidanand Vaidya <email address hidden>, Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Sachin,

I triggered the sanity in the following container and its running now

[root@a3s10 ~]# docker attach ab0e1c12eb88

[root@ab0e1c12eb88 contrail-test]# bash -x run_tests.sh  -T vcenter --send-mail -U



2018-08-18 09:41:56,829 - INFO - ================================================================================

2018-08-18 09:41:56,829 - INFO - STARTING TEST    : test_contrail_alarms

2018-08-18 09:41:56,831 - INFO - TEST DESCRIPTION :  Test to check if alarms are present





2018-08-18 09:42:00,114 - INFO - Initial checks done. Running the testcase now

2018-08-18 09:42:00,115 - INFO -

2018-08-18 09:42:03,333 - INFO - END TEST : test_contrail_alarms : PASSED[0:00:07]

2018-08-18 09:42:03,334 - INFO - --------------------------------------------------------------------------------

2018-08-18 09:42:03,339 - INFO - ================================================================================

2018-08-18 09:42:03,339 - INFO - STARTING TEST    : test_contrail_status

2018-08-18 09:42:03,339 - INFO - TEST DESCRIPTION :  Test to verify that all services are running and active



Thanks,
Aswani Kumar

From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, 18 August 2018 at 10:03 PM
To: Madhava Rao Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Ok. Single Sanity run passed.
2018-08-18 08:53:06,766 - INFO - END TEST : test_ping_within_vn : PASSED[0:26:42]
2018-08-18 08:53:06,766 - INFO - --------------------------------------------------------------------------------

But I’m trying to run complete Sanity using following command:
[root@ab0e1c12eb88 contrail-test]# ./run_tests.sh -T venter --send-mail -

The run was over immediately..
Sandip/ Sudhee, can you pls let us what the error is here ?

I have attached complete log to the mail. Relevant logs are included :

[root@10.84.16.10] sudo: systemctl enable nfs-server.service
[root@10.84.16.10] sudo: systemctl start nfs-server.service
[root@10.84.16.11] run: vim-cmd hostsvc/vmotion/vnic_set vmk0
[root@10.84.16.12] run: vim-cmd hostsvc/vmotion/vnic_set vmk0
Exception in thread Thread-8 (most likely raised during interpreter shutdown):Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
[localhost] local: rpm -q --queryformat '%{VERSION}-' contrail-test; rpm -q --queryformat '%{RELEASE}' contrail-test | awk -F'.' '{print $1}'
Exception in thread Thread-3 (most likely raised during interpreter shutdown):Generated report_details* file: report_details_2018_08_18_09_29_40.ini
could not change test cases names
could not change test cases names
Buildfile: /contrail-test/build.xml

junit-report:
   [delete] Deleting directory /contrail-test/report
[junitreport] Processing /contrail-test/report/TESTS-TestSuites.xml to /contrail-test/report/junit-noframes.html
[junitreport] Loading stylesheet /contrail-test/junit-noframes.xsl
[junitreport] Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
[junitreport] Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.
[junitreport] Transform time: 324ms

BUILD SUCCESSFUL
Total time: 0 seconds
Generate HTML reports in report/ folder : report/junit-noframes.html
logs/tcutils.util.log
logs/tcutils.config.vcenter_verification.log
logs/__main__.log
logs/tcutils.collector.opserver_results.log
logs/introspect.log
logs/a3s10-tracebacks.log
logs/a3s10-ignored_tracebacks.log
logs/a3s10-traceback_parse_failures.log
logs/common.contrail_test_init.log
logs/testbasicvmvn.log
Converted log files to html files
[root@10.84.16.10] put: tools/search-bt.py -> /tmp/search-bt.py
Traceback details will be in /tmp/a3s10-tracebacks.log
Ignored Traceback details will be in /tmp/a3s10-ignored_tracebacks.log
Traceback parse errors will be in /tmp/a3s10-traceback_parse_failures.log
Done looking for tracebacks
[root@10.84.16.10] download: /contrail-test/logs/a3s10-tracebacks.log <- /tmp/a3s10-tracebacks.log

Warning: Local file /contrail-test/logs/a3s10-tracebacks.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-tracebacks.log'

Underlying exception:
    No such file

[root@10.84.16.10] download: /contrail-test/logs/a3s10-ignored_tracebacks.log <- /tmp/a3s10-ignored_tracebacks.log

Warning: Local file /contrail-test/logs/a3s10-ignored_tracebacks.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-ignored_tracebacks.log'

Underlying exception:
    No such file

[root@10.84.16.10] download: /contrail-test/logs/a3s10-traceback_parse_failures.log <- /tmp/a3s10-traceback_parse_failures.log

Warning: Local file /contrail-test/logs/a3s10-traceback_parse_failures.log already exists and is being overwritten.


Warning: get() encountered an exception while downloading '/tmp/a3s10-traceback_parse_failures.log'

Underlying exception:
    No such file

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Uploaded reports
Not all mail server details are available. Skipping mail send.
Sent mail to interested parties
[root@ab0e1c12eb88 contrail-test]#


Thanks,
Sachin
From: Sachchidanand Vaidya <email address hidden>
Date: Saturday, August 18, 2018 at 8:30 AM
To: Sudheendra Rao <email address hidden>, Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

After removing nfs-ds from ESXI hosts datastore, test is running now.
I will post the result once the tests are run.

Thanks
Sachin

From: Sudheendra Rao <email address hidden>
Date: Saturday, August 18, 2018 at 4:15 AM
To: Ram Yadav <email address hidden>, Sandip Dey <email address hidden>, Aswani Kumar Gaddam <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Marcin Orkiszewski <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Hi Ram,

    Sandip had already mentioned that automation code expects that vcenter_vars.yaml has ‘server1’ only under ‘vcenter_servers’, anything else will result in error.

Change all occurrence of SRV1 in vcenter_vars.yaml to server1 and launch new test container using test_runner.sh
I tried and it got past that stage, but seeing below error, I don’t understand this, please check – the test container is ab0e1c12eb88

[root@ab0e1c12eb88 contrail-test]# python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn
:
:

[root@10.84.16.10] out: Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@10.84.16.10] out:

[root@10.84.16.10] sudo: systemctl start nfs-server.service
======================================================================
ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass
    super(TestBasicVMVN, cls).setUpClass()
  File "scripts/vm_regression/base.py", line 12, in setUpClass
    super(BaseVnVmTest, cls).setUpClass()
  File "test_v1.py", line 38, in setUpClass
    cls.admin_isolated_creds.setUp()
  File "common/isolated_creds.py", line 132, in setUp
    password=self.password)
  File "common/connections.py", line 68, in __init__
    self.vnc_lib = self.get_vnc_lib_h()
  File "common/connections.py", line 167, in get_vnc_lib_h
    env[attr].setUp()
  File "/contrail-test/fixtures/vnc_api_test.py", line 113, in setUp
    if self.orch:
  File "/contrail-test/fixtures/vnc_api_test.py", line 178, in orch
    self._orch = self.get_orch_h()
  File "/contrail-test/fixtures/vnc_api_test.py", line 201, in get_orch_h
    logger=self.logger)
  File "/contrail-test/fixtures/vcenter.py", line 144, in __init__
    self._nfs_ds = NFSDatastore(self._inputs, self)
  File "tcutils/util.py", line 997, in __call__
    Singleton, cls).__call__(*args, **kwargs)
  File "/contrail-test/fixtures/vcenter.py", line 92, in __init__
    host.configManager.datastoreSystem.CreateNasDatastore(spec)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1366, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
vim.fault.DuplicateName: (vim.fault.DuplicateName) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = "The name 'nfs-ds' already exists.",
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   name = 'nfs-ds',
   object = 'vim.Datastore:10.84.16.10:/nfs'
}

Ran 0 tests in 25.285s
FAILED (failures=1)


Thanks,
Sudhee.

From: Ram Yadav <email address hidden>
Date: Saturday, 18 August 2018 at 6:21 AM
To: Sandip Dey <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Sandip Dey <email address hidden>, Marcin Orkiszewski <email address hidden>, Sudheendra Rao <email address hidden>, Adrian Szczepański <email address hidden>, Amudha R <email address hidden>
Subject: Re: steps to run sanity on vCenter local setup's

Hi Sandip,
I pulled in the docker image:
contrail-test-test:master-221

And followed the steps for single sanity test case run.
I’m getting the following error:

ERROR: setUpClass (scripts.vm_regression.test_vm_basic.TestBasicVMVN)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "scripts/vm_regression/test_vm_basic.py", line 27, in setUpClass

    super(TestBasicVMVN, cls).setUpClass()

  File "scripts/vm_regression/base.py", line 12, in setUpClass

    super(BaseVnVmTest, cls).setUpClass()

  File "test_v1.py", line 13, in setUpClass

    super(BaseTestCase_v1, cls).setUpClass()

  File "test.py", line 79, in setUpClass

    cls.inputs = ContrailTestInit(cls.input_file,logger = cls.logger)

  File "common/contrail_test_init.py", line 853, in __init__

    self.inputs = TestInputs(input_file, self.logger)

  File "tcutils/util.py", line 997, in __call__

    Singleton, cls).__call__(*args, **kwargs)

  File "common/contrail_test_init.py", line 102, in __init__

    self.parse_yml_file()

  File "common/contrail_test_init.py", line 589, in parse_yml_file

    self.vcenter_dc = _parse_vcenter.vcenter_dc

  File "common/contrail_test_init.py", line 1466, in vcenter_dc
















Test container is spawned on 10.84.16.10.

/root/contrail-ansible-deployer has the contrail_test_input.yml and vcenter_vars.yml









Thanks,

Ram








From: Ram Yadav <<email address hidden><mailto:<email address hidden>>>
Date: Tuesday, August 14, 2018 at 4:31 PM
To: Adrian Szczepański <<email address hidden><mailto:<email address hidden>>>, Amudha R <<email address hidden><mailto:<email address hidden>>>
Cc: Sachchidanand Vaidya <<email address hidden><mailto:<email address hidden>>>, Sandip Dey <<email address hidden><mailto:<email address hidden>>>, Marcin Orkiszewski <<email address hidden><mailto:<email address hidden>>>
Subject: steps to run sanity on vCenter local setup's

Hello,

Below is the instruction to run sanity

1.wget https://github.com/Juniper/contrail-test/raw/master/testrunner.sh
2.docker pull 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
3../testrunner.sh run -s -P contrail_test_input.yml,vcenter_vars.yml 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
4.cd contrail-test
5.export TEST_CONFIG_FILE=contrail_test_input_useit.yaml
6./run_tests.sh -T venter --send-mail -











To run a single test case
1.wget https://github.com/Juniper/contrail-test/raw/master/testrunner.sh
2.docker pull 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
3../testrunner.sh run -s -P contrail_test_input.yml,vcenter_vars.yml 10.204.217.152:5000/contrail-test-test:ocata-5.0-<build-id>
4.cd contrail-test
5.export PYTHONPATH=$PATH:$PWD:$PWD/fixtures:$PWD/scripts
6.export TEST_CONFIG_FILE=contrail_test_input_useit.yaml
7.python -m testtools.run scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn

Docker pull needs to be done on ci-repo.englab.juniper.net or local repo as mentioned in the above step which is Bangalore local repo.

Apart from this you will need set of vmdk file which was copied by Sandip Dey at some sunnyvale location.

Sandip,
Can provide detail on the file’s that we need to copy say to codilime’s server locally for sanity to run on there setup?

Thanks,
Ram