snat/lbaas SIs failed to get launched in sanity

[Vedamurthy Joshi]

R2.20 Build 60 Ubuntu 12.04 Icehouse

Snat and lbaas sanity tests failed : http://10.204.216.50/Docs/logs/2.20-60_2015_06_21_23_58_20/junit-noframes.html

Logs : /cs-shared/test_runs/a6s32/2015_06_21_23_58_20
http://10.204.216.50/Docs/logs/2.20-60_2015_06_21_23_58_20/logs/testroutersnat.log

It was seen that netns instances were not launched on the vrouter. "Template not found " errors were seen in svc monitor logs as shown below

contrail-svc-monitor.log :
=======================

06/21/2015 12:34:58 PM [a6s32:contrail-svc-monitor:Config:0]: Starting Introspect on HTTP Port 8088
06/21/2015 12:34:58 PM [a6s32:contrail-svc-monitor:Config:0]: Cannot write http_port 8088 to /tmp/contrail-svc-monitor.16079.http_port
06/21/2015 12:49:18 PM [a6s32:contrail-svc-monitor:Config:0]: __default__ [SYS_ERR]: SvcMonitorLog: nova error Port 3e98ac9e-c8eb-4f63-a7c7-a6a48fdd1fe9 is still in use. (HTTP 409) (Request-ID: req-fb51342a-7ea5-4b04-97ea-08c4f33ff380)
06/21/2015 12:49:49 PM [a6s32:contrail-svc-monitor:Config:0]: __default__ [SYS_ERR]: SvcMonitorLog: nova error Port db057dd6-20d1-4cde-a383-009c74c674d2 is still in use. (HTTP 409) (Request-ID: req-4bfdf817-b7f4-4504-94b8-c2c19ed7e490)
06/21/2015 12:52:58 PM [a6s32:contrail-svc-monitor:Config:0]: __default__ [SYS_ERR]: SvcMonitorLog: template not found for default-domain:TestLbaas-40547969:16c6b177-6170-48bd-aa5e-314a0133c664
06/21/2015 12:54:25 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPwithSVMChange-81198436:bridge_si-10065476_1>
06/21/2015 12:54:26 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPwithSVMChange-81198436:bridge_si-10065476_1>
06/21/2015 12:54:27 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPwithSVMChange-81198436:bridge_si-10065476_1>
06/21/2015 12:57:55 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPSanity-97341809:in_net_svc_instance-79958094_1>
06/21/2015 12:57:55 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPSanity-97341809:in_net_svc_instance-79958094_1>
06/21/2015 01:05:24 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPSanity-97341809:bridge_si-11034403_1>
06/21/2015 01:05:25 PM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestECMPSanity-97341809:bridge_si-11034403_1>
06/21/2015 01:19:32 PM [a6s32:contrail-svc-monitor:Config:0]: __default__ [SYS_ERR]: SvcMonitorLog: template not found for default-domain:TestRouterSNAT-80670337:si_215028dd-6ec5-47c1-acc8-bcccf56db2eb
06/21/2015 02:39:40 PM [a6s32:contrail-svc-monitor:Config:0]: Starting Introspect on HTTP Port 8088
06/21/2015 02:39:40 PM [a6s32:contrail-svc-monitor:Config:0]: Cannot write http_port 8088 to /tmp/contrail-svc-monitor.29354.http_port
06/22/2015 12:35:45 AM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestLbaas-85362737:afeed3f3-7be7-4c9b-aad3-1f8073e6a84f>
06/22/2015 12:39:18 AM [a6s32:contrail-svc-monitor:Config:0]: Duplicate uve delete <default-domain:TestLbaas-31065321:9a48f122-fbb6-4f44-b0ce-3b5ab6e714a3>

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11931
Submitter: Prakash Bailkeri (<email address hidden>)

[Prakash Bailkeri (prakashmb)]

Following traceback is also found in svc mon log.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 792, in launch_timer
    timer_callback(monitor)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 768, in timer_callback
    if not monitor._check_service_running(si):
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 724, in _check_service_running
    if st.virtualization_type == 'virtual-machine':
AttributeError: 'NoneType' object has no attribute 'virtualization_type'

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11931
Committed: http://github.org/Juniper/contrail-controller/commit/37f9ede88e573a9cd996bf391af0294e1c0ee5c4
Submitter: Zuul
Branch: R2.20

commit 37f9ede88e573a9cd996bf391af0294e1c0ee5c4
Author: Prakash Bailkeri <email address hidden>
Date: Mon Jun 22 06:04:49 2015 -0700

default service template is not added to local config db on first time create

Fix: Fetch the default service template created by svc-mon explicitly

Change-Id: Ie1a89a22bf3a965707ade0d4b8ead243b80d548b
Closes-bug: #1467435

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11992
Submitter: Prakash Bailkeri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11992
Committed: http://github.org/Juniper/contrail-controller/commit/bce214140690dda463eae18aa5911587fa05b1d3
Submitter: Zuul
Branch: master

commit bce214140690dda463eae18aa5911587fa05b1d3
Author: Prakash Bailkeri <email address hidden>
Date: Mon Jun 22 06:04:49 2015 -0700

default service template is not added to local config db on first time create

Fix: Fetch the default service template created by svc-mon explicitly

Change-Id: Ie1a89a22bf3a965707ade0d4b8ead243b80d548b
Closes-bug: #1467435