CI Tempest failures seen quite frequently in ubuntu precise icehouse systest job

[Ananth Suryanarayana]

In the past few days, multiple jobs have failed in Tempest tests. Initial analysis from Sachin and Ananth did not yield any clue yet.
This bug is to track this particular tempest failures in CI

e.g.

https://review.opencontrail.org/10476 8:53 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6813/
FAILURE same as with previous https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6787/

https://review.opencontrail.org/10526 9:08 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6799/

https://review.opencontrail.org/10511 9:36 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6821/
FAILURE same as with previous https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6798/

https://review.opencontrail.org/10518 9:37 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6818/
FAILURE same as with previous https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6786/

5: Tempest and Sanity failures

Tempest tests FAILED
Sanity tests that failed:
TestECMPFeature.test_ecmp_svc_in_network_nat_with_3_instance
TestBasicPolicyConfig.test_policy
TestSvcRegr.test_svc_monitor_datapath
TestBasicVMVN6.test_metadata_service
https://review.opencontrail.org/10346 10:23 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6809/
http://10.204.216.50/Docs/logs/2.20-1323_2015-05-19_08:39:14/junit-noframes.html

Tempest tests FAILED
Sanity tests passed but reported "ci_sanity FAILED"
https://review.opencontrail.org/10477 10:34 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6808/
http://10.204.216.50/Docs/logs/2.20-8450_2015-05-19_08:37:23/junit-noframes.html

Tempest tests FAILED
ci_sanity FAILED -- ALL 21 FAILED
https://review.opencontrail.org/10537 10:43 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6803/

Tempest tests FAILED
Sanity tests passed but reported "ci_sanity FAILED" -- one skipped, rest passed
TestRouterSNAT.test_basic_snat_behavior_without_external_connectivity[ci_sanity] Skipping Test. Env variable MX_GW_TEST is not set
https://review.opencontrail.org/10538 10:43 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6804/

Tempest tests FAILED: tempest.exceptions.BuildErrorException: Server
Sanity tests passed but reported "ci_sanity FAILED" -- one skipped, rest passed
TestRouterSNAT.test_basic_snat_behavior_without_external_connectivity[ci_sanity] Skipping Test. Env variable MX_GW_TEST is not set
https://review.opencontrail.org/10063 10:47 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6822/

Tempest tests FAILED
Sanity tests that failed:
SecurityGroupRegressionTests1.test_sec_group_add_delete
https://review.opencontrail.org/10536 10:56 AM
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6802/
http://10.204.216.50/Docs/logs/3.0-2816_2015-05-19_09:24:37/junit-noframes.html

[Vinay Mahuli (vmahuli)]

Some more from my yesterday's triage.

2. Tempest test failed:
http://10.204.216.50/Docs/logs/ci-tempest/result-10538,1-1952015212310.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6853/console
https://review.opencontrail.org/#/c/10538/

http://10.204.216.50/Docs/logs/ci-tempest/result-10582,1-1952015174359.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6844/console
https://review.opencontrail.org/#/c/10582/

http://10.204.216.50/Docs/logs/ci-tempest/result-10511,2-1952015175016.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6845/console
https://review.opencontrail.org/#/c/10511/

http://10.204.216.50/Docs/logs/ci-tempest/result-10526,1-1952015214122.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6854/console
https://review.opencontrail.org/#/c/10526/

http://10.204.216.50/Docs/logs/ci-tempest/result-10579,1-1952015171737.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6842/console
https://review.opencontrail.org/#/c/10579/

http://10.204.216.50/Docs/logs/ci-tempest/result-10476,1-1952015201243.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6849/console
https://review.opencontrail.org/#/c/10476/

http://10.204.216.50/Docs/logs/ci-tempest/result-10571,1-19520151583.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6833/console
https://review.opencontrail.org/#/c/10571/

http://10.204.216.50/Docs/logs/ci-tempest/result-10574,1-1952015152838.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6835/console
https://review.opencontrail.org/#/c/10574/

http://10.204.216.50/Docs/logs/ci-tempest/result-10517,1-195201516472.xml
https://jenkins.opencontrail.org/job/ci-contrail-controller-systest-ubuntu-precise-pangolin-icehouse/6841/console
https://review.opencontrail.org/#/c/10517/

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11983
Submitter: Praneet Bachheti (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11984
Submitter: Praneet Bachheti (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11984
Committed: http://github.org/Juniper/contrail-controller/commit/03ab52c41e733d25631a5457640854b1165760e4
Submitter: Zuul
Branch: master

commit 03ab52c41e733d25631a5457640854b1165760e4
Author: Praneet Bachheti <email address hidden>
Date: Tue Jun 23 16:50:14 2015 -0700

if resource exists, do not overwrite id_perms

Change-Id: If907c2780a50632951ca1dc8147cd81e51a65920
Partial-Bug: #1457227

[Hampapur Ajay (hajay)]

The review/commit in https://review.opencontrail.org/11984 addressed the fact that in _http_put_common there could be a race-condition leading to read-modify-write in id-perms values that would clobber the description that tempest SG test was checking for.

For instance:
1. tempest test created SG
2. schema transformer updated SG to update SG-id. The _put_common would read in id-perms
3. tempest test updated SG description
4. 2. finishes and clobbers SG description
5. tempest test reads back SG finds no description and fails.

see attached file for actual scenario.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11983
Committed: http://github.org/Juniper/contrail-controller/commit/0c61acd823889016f94995b742c6558667125373
Submitter: Zuul
Branch: R2.20

commit 0c61acd823889016f94995b742c6558667125373
Author: Praneet Bachheti <email address hidden>
Date: Tue Jun 23 16:52:28 2015 -0700

if resource exists, do not overwrite id_perms

Change-Id: I9c07d2d2da2b5632792345685a5d98df0d32f27b
Partial-Bug: #1457227

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13927
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14196
Submitter: Hampapur Ajay (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14196
Committed: http://github.org/Juniper/contrail-controller/commit/559056d1af967a3f3f0bdcc90877f8333ead9783
Submitter: Zuul
Branch: R2.20.x

commit 559056d1af967a3f3f0bdcc90877f8333ead9783
Author: Praneet Bachheti <email address hidden>
Date: Tue Jun 23 16:50:14 2015 -0700

if resource exists, do not overwrite id_perms

Change-Id: If907c2780a50632951ca1dc8147cd81e51a65920
Partial-Bug: #1457227
(cherry picked from commit 03ab52c41e733d25631a5457640854b1165760e4)