[R5.0]- Fix intermittent failure for test cases test_flow_action_drop_stats

[alok kumar]

Test case TestDropStats.test_flow_action_drop_stats fails intermittently in sanity with below traceback:

Traceback (most recent call last): File "tcutils/wrappers.py", line 78, in wrapper result = function(self, *args, **kwargs) File "serial_scripts/vrouter/test_drop_stats.py", line 38, in test_flow_action_drop_stats assert self.verify_flow_action_drop_stats() File "common/vrouter/base_drop_stats.py", line 58, in verify_flow_action_drop_stats vif_dict_before = self.get_drop_stats_dict(compute0, fq_name) File "common/vrouter/base_drop_stats.py", line 32, in get_drop_stats_dict drop_stats_list = self.agent_inspect_h[compute].get_agent_vm_interface_drop_stats(fq_name) File "tcutils/agent/vna_introspect_utils.py", line 876, in get_agent_vm_interface_drop_stats raw_drop_stats_obj = raw_drop_stats_obj[index] IndexError: list index out of range

though this is not reproducible in current setup, this probably looks like script issue when raw_drop_stats field is not available in VMI table in agent.
All the fields for VMI get updated periodically and there may be instances when VMI table is seen but field raw_drop_stats is not updated yet.
In this case we may need to retry.

[alok kumar (kalok)]

In Failure case, below was the query and response from agent introspect where field raw_drop_stats is missing in the response as after new VM launch agent takes sometime to query vrouter to update this field. This field is optional in agent VMI uve, so it will not be initially present when VM is launched.

2018-06-14 12:51:37,308 - DEBUG - Requesting: http://10.204.216.99:8085/Snh_SandeshUVECacheReq?x=UveVMInterfaceAgent
2018-06-14 12:51:37,313 - DEBUG - <__UveVMInterfaceAgentTrace_list type="slist">
  <UveVMInterfaceAgentTrace type="sandesh">
    <data type="struct" identifier="1">
      <UveVMInterfaceAgent>
        <name type="string" identifier="1" key="ObjectVMITable">default-domain:ctest-TestDropStats-19689257:8d4e4f4e-051d-4521-b17e-74d1d8e3e0ad</name>
        <ip_address type="string" identifier="3">68.225.53.3</ip_address>
        <virtual_network type="string" identifier="4">default-domain:ctest-TestDropStats-19689257:ctest-vn-69421443</virtual_network>
        <mac_address type="string" identifier="5">02:8d:4e:4f:4e:05</mac_address>
        <label type="u32" identifier="6">25</label>
        <active type="bool" identifier="7">true</active>
        <vm_name type="string" identifier="8">ctest-vm1-36599244</vm_name>
        <gateway type="string" identifier="9">68.225.53.1</gateway>
        <l2_active type="bool" identifier="10">true</l2_active>
        <uuid type="string" identifier="11">8d4e4f4e-051d-4521-b17e-74d1d8e3e0ad</uuid>
        <ip6_address type="string" identifier="12">::</ip6_address>
        <ip6_active type="bool" identifier="13">false</ip6_active>
        <floating_ips type="list" identifier="15">
          <list type="struct" size="0"/>
        </floating_ips>
        <vm_uuid type="string" identifier="19">bd912d8e-af4c-44c1-bdf1-0177fc435a16</vm_uuid>
        <ip4_active type="bool" identifier="22">true</ip4_active>
        <is_health_check_active type="bool" identifier="23">true</is_health_check_active>
        <health_check_instance_list type="list" identifier="24">
          <list type="struct" size="0"/>
        </health_check_instance_list>
        <admin_state type="bool" identifier="27">true</admin_state>
        <fixed_ip4_list type="list" identifier="28">
          <list type="string" size="1">
            <element>68.225.53.3</element>
          </list>
        </fixed_ip4_list>
        <fixed_ip6_list type="list" identifier="29">
          <list type="string" size="0"/>
        </fixed_ip6_list>
        <vn_uuid type="string" identifier="37">1db491b7-0f31-4e52-a022-dfca33759b8f</vn_uuid>
        <tx_vlan type="u16" identifier="38">65535</tx_vlan>
        <rx_vlan type="u16" identifier="39">65535</rx_vlan>
        <vhostuser_mode type="byte" identifier="49">0</vhostuser_mode>
        <port_mirror_enabled type="bool" identifier="50">false</port_mirror_enabled>
      </UveVMInterfaceAgent>
    </data>
  </UveVMInterfaceAgentTrace>
  <UveVMInterfaceAgentTrace type="sandesh">
    <data type="struct" identifier="1">
      <UveVMInterfaceAgent>
        <name type="string" identifier="1" key="ObjectVMITable">default-global-system-config:nodem10:vhost0</name>
        <ip_address type="string" identifier="3">10.10....

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43983
Submitter: alok kumar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44111
Submitter: alok kumar (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/43983
Committed: http://github.com/Juniper/contrail-test/commit/10f774e1c2667c501089a8cef6f7eca8cee1d27c
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 10f774e1c2667c501089a8cef6f7eca8cee1d27c
Author: Alok Kumar <email address hidden>
Date: Wed Jun 20 15:05:14 2018 +0530

Enabling introspect ssl tests to run in microservices setup

1. Update common.sh in place of config file of
 contrail services
2. Optimize method add_knob_to_container
3. Optimizing few methods to verify service
 status with ssl certs
4. Fix for intermittent failure of test_flow_action_drop_stats

Change-Id: I2ac282e8f2475fc38280dab8783043632b6c8228
Closes-bug: #1777107
Partial-bug: #1777106

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44111
Committed: http://github.com/Juniper/contrail-test/commit/0b984264b2939f41a2be2d5c2af3b2e0b9f0cf77
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 0b984264b2939f41a2be2d5c2af3b2e0b9f0cf77
Author: Alok Kumar <email address hidden>
Date: Wed Jun 20 15:05:14 2018 +0530

Enabling introspect ssl tests to run in microservices setup

1. Update common.sh in place of config file of
 contrail services
2. Optimize method add_knob_to_container
3. Optimizing few methods to verify service
 status with ssl certs
4. Fix for intermittent failure of test_flow_action_drop_stats

Change-Id: I2ac282e8f2475fc38280dab8783043632b6c8228
Closes-bug: #1777107
Partial-bug: #1777106