Sanity: VM delete verification asserts due to VM entry seen in agent

[Sarath]

This issue seen on Vcenter-compute sanity on multiples of testcases failing listed below,
test_ping_within_vn
test_vm_add_delete
test_control_node_switchover

>>> below is the failure logs

2017-07-20 01:55:37,780 - INFO - Deleting VM ctest-vm1-06332580

2017-07-20 01:55:43,537 - INFO - VM ctest-vm1-06332580 is fully removed in API-Server

2017-07-20 01:55:43,618 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:45,644 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:47,674 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:49,703 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:51,733 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:53,762 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:55,789 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:57,819 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:55:59,845 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:01,876 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:03,904 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:05,931 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:07,969 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:09,995 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:12,017 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:14,043 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:16,068 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:18,094 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:20,122 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:22,149 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:24,173 - WARNING - VM ctest-vm1-06332580 is still present in agent's active VMList

2017-07-20 01:56:24,193 - INFO - Deleting VN ctest-vn30-40338861

2017-07-20 01:56:24,538 - INFO - Validated that VN ctest-vn30-40338861 is not found in API Server

2017-07-20 01:56:24,562 - INFO - Validated that VN ctest-vn30-40338861 is not in any agent

2017-07-20 01:56:24,585 - INFO - Validated that Routing instances and Config db in Control-nodes does not have VN ctest-vn30-40338861 info

2017-07-20 01:56:33,565 - ERROR -

  File "tcutils/wrappers.py", line 95, in wrapper

    cleanup(*args, **kwargs)

  File "/contrail-test/fixtures/vm_test.py", line 1884, in cleanUp

    self.delete()

  File "/contrail-test/fixtures/vm_test.py", line 1910, in delete

    self.verify_cleared_from_setup(verify=verify)

  File "/contrail-test/fixtures/vm_test.py", line 1925, in verify_cleared_from_setup

    'one or more agents' % (self.vm_name))

AssertionError:

VM ctest-vm1-06332580 is still seen in one or more agents

2017-07-20 01:56:33,565 - INFO -

2017-07-20 01:56:33,565 - INFO - END TEST : test_ping_within_vn : FAILED[0:02:47]

2017-07-20 01:56:33,565 - INFO - --------------------------------------------------------------------------------

�/�@�YppA���@~scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn[ci_sanity,quick_sanity,sanity,suite1,vcenter,vrouter_gw]
worker-1text/plain; charset="utf8"pythonlogging:''
1_����/�@�YppA���@~scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn[ci_sanity,quick_sanity,sanity,suite1,vcenter,vrouter_gw]
worker-13text/x-traceback; charset="utf8"; language="python" traceback#Traceback (most recent call last):

1V�O��/�A9YppA���@~scripts.vm_regression.test_vm_basic.TestBasicVMVN.test_ping_within_vn[ci_sanity,quick_sanity,sanity,suite1,vcenter,vrouter_gw]
worker-13text/x-traceback; charset="utf8"; language="python" traceback@\ File "tcutils/wrappers.py", line 152, in wrapper

    raise TestFailed("\n ".join(errmsg))

[Sarath (nsarath)]

Analysis: Verified on indivdual test-runs, it pass prooving timing issue. Increasing timer from 5 to 15sec

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/33910
Submitter: Sarath (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/33910
Submitter: Senthilnathan Murugappan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/33910
Committed: http://github.com/Juniper/contrail-test-ci/commit/aa254bbff49957d60352c5432aa6101669490137
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit aa254bbff49957d60352c5432aa6101669490137
Author: sarath <email address hidden>
Date: Fri Jul 21 16:44:14 2017 -0700

fix timer for unconfigure failure issue

test_ping_within_vn
test_vm_add_delete
test_control_node_switchover scripts unconfigure issues seen reporting entries still present

Change-Id: Ia590d277ac47121eb5b589af528b618122f734ab
Closes-Bug: #1705788

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34139
Submitter: Senthilnathan Murugappan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34140
Submitter: Senthilnathan Murugappan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34139
Committed: http://github.com/Juniper/contrail-test-ci/commit/031a661aa864de8a4349847bf4245a9cabac0b56
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 031a661aa864de8a4349847bf4245a9cabac0b56
Author: sarath <email address hidden>
Date: Fri Jul 21 16:44:14 2017 -0700

fix timer for unconfigure failure issue

test_ping_within_vn
test_vm_add_delete
test_control_node_switchover scripts unconfigure issues seen reporting entries still present

Change-Id: Ia590d277ac47121eb5b589af528b618122f734ab
Closes-Bug: #1705788
(cherry picked from commit aa254bbff49957d60352c5432aa6101669490137)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34140
Committed: http://github.com/Juniper/contrail-test-ci/commit/9a5b8d8b9df4393242811de9e116b0a684b3de74
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9a5b8d8b9df4393242811de9e116b0a684b3de74
Author: sarath <email address hidden>
Date: Fri Jul 21 16:44:14 2017 -0700

fix timer for unconfigure failure issue

test_ping_within_vn
test_vm_add_delete
test_control_node_switchover scripts unconfigure issues seen reporting entries still present

Change-Id: Ia590d277ac47121eb5b589af528b618122f734ab
Closes-Bug: #1705788
(cherry picked from commit aa254bbff49957d60352c5432aa6101669490137)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35718
Submitter: Sarath (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36200
Submitter: Sarath (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/36201
Submitter: Sarath (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36200
Committed: http://github.com/Juniper/contrail-test-ci/commit/7868e16e4d6926075d950fadc1dc6a239f6ba8ee
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 7868e16e4d6926075d950fadc1dc6a239f6ba8ee
Author: sarath <email address hidden>
Date: Mon Sep 18 23:56:00 2017 -0700

to verify ping for 3 vm so that covers vcenter-compute clusters scenario

Closes-Bug: #1705788

Change-Id: Ibd3e0eb153664d58e438e362af9f0cf346fc9a50

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/36201
Committed: http://github.com/Juniper/contrail-test-ci/commit/a077a1abe8717a2149dc16a3f4d4fefdbd5587f7
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit a077a1abe8717a2149dc16a3f4d4fefdbd5587f7
Author: sarath <email address hidden>
Date: Mon Sep 18 23:56:00 2017 -0700

to verify ping for 3 vm so that covers vcenter-compute clusters scenario

Closes-Bug: #1705788

Change-Id: Ibd3e0eb153664d58e438e362af9f0cf346fc9a50

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/40095
Submitter: Ankit Jain (<email address hidden>)