[4.1.0.0-44] Internal RIs not created in a ECMP-SVC case.

[Ganesha HV]

Failing sanity testcase:
=========================
scripts.ecmp.test_ecmp.TestECMPFeature.test_ecmp_svc_v2_in_network_nat_with_3_instance

1]. Created a in-network-nat ECMP SVC between ctest-left_ctest-TestECMPFeature-72816639-24666446
(59.55.203.0/24) and ctest-right_ctest-TestECMPFeature-72816639-52542641(114.201.64.0/24).

2]. None of the Internal RIs are created and the SVC is not working.

3]. The following tracebacks are seen in the contrail-api log:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1578, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_api_stats.py", line 17, in wrapper
    response = func(api_server_obj, resource_type, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1039, in http_resource_list
    pagination)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 2960, in _list_collection
    paginate_count=page_count)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_db.py", line 1309, in dbe_list
    paginate_count=paginate_count)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 492, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 1173, in object_list
    children_fq_names_uuids.extend(filter_rows_backref_anchor())
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 1157, in filter_rows_backref_anchor
    for cols in obj_rows[obj_key].items()]
AttributeError: 'NoneType' object has no attribute 'items'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 1073, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 504, in plugin_http_post_floatingip
    return self.plugin_get_floatingips(context, floatingip)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 473, in plugin_get_floatingips
    floatingips_info = cfgdb.floatingip_list(context, filters)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 2818, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3926, in floatingip_list
    detail=True)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 627, in _objects_list
    detail=detail, count=count, filters=filters, shared=shared)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 1314, in resource_list
    rest.OP_GET, obj_class.create_uri, data=query_params)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 855, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/context.py", line 37, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 116, in _request
    ret_val = server_method()
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/context.py", line 122, in wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1578, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_api_stats.py", line 17, in wrapper
    response = func(api_server_obj, resource_type, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1039, in http_resource_list
    pagination)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 2960, in _list_collection
    paginate_count=page_count)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_db.py", line 1309, in dbe_list
    paginate_count=paginate_count)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 492, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 1173, in object_list
    children_fq_names_uuids.extend(filter_rows_backref_anchor())
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 1157, in filter_rows_backref_anchor
    for cols in obj_rows[obj_key].items()]
AttributeError: 'NoneType' object has no attribute 'items'

[Ignatious Johnson Christopher (ijohnson-x)]

WARNING:contrail-api:__default__ [SYS_WARN]: VncApiError: Multiget result contains a key (d96cd6c4-46a3-4ca6-9ccf-565169ae1f46) with an empty value. obj_uuid_table: keys ([u'4676db78-b816-4a13-bda5-6faa985d26ad', u'dd95f629-9730-42fe-866a-5e588a26acd8', u'd96cd6c4-46a3-4ca6-9ccf-565169ae1f46']), columns (None), start (backref:virtual_machine_interface:), finish (backref:virtual_machine_interface;)
WARNING:contrail-api:__default__ [SYS_WARN]: VncApiError: Multiget result contains a key (dd95f629-9730-42fe-866a-5e588a26acd8) with an empty value. obj_uuid_table: keys ([u'4676db78-b816-4a13-bda5-6faa985d26ad', u'dd95f629-9730-42fe-866a-5e588a26acd8', u'd96cd6c4-46a3-4ca6-9ccf-565169ae1f46']), columns (None), start (backref:virtual_machine_interface:), finish (backref:virtual_machine_interface;)
ERROR:contrail-api:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.AttributeError'>
Python 2.7.6: /usr/bin/python
Sat Sep 9 09:15:38 2017

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35443
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35443
Committed: http://github.com/Juniper/contrail-controller/commit/f2009d51bf532e1756b051a1c9f485e1101c254c
Submitter: Zuul (<email address hidden>)
Branch: master

commit f2009d51bf532e1756b051a1c9f485e1101c254c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Sat Sep 9 14:04:20 2017 -0700

delete the row keys from the multiget results if

the value is empty for the key.

Change-Id: I1f2148dd49eab1e1e8b807309e0f4b49001a06d3
Closes-Bug: 1715176