VNC API server fails to read a resource with more that 10k ref/back_ref

[Édouard Thuleau]

During scaling tests where more than 10k network/subnet are created, the VNC server APi fails to show the default network-ipam resource with that traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1563, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_api_stats.py", line 17, in wrapper
    response = func(api_server_obj, resource_type, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 559, in http_resource_read
    (ok, result) = db_conn.dbe_read(resource_type, obj_ids, obj_fields)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1784, in dbe_read
    obj_type, [obj_ids['uuid']], obj_fields)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 523, in object_read
    include_timestamp=True)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 270, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/columnfamily.py", line 772, in multiget
    packed_keys[offset:offset + buffer_size], cp, sp, consistency)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 669, in multiget_slice
    return self.recv_multiget_slice()
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 690, in recv_multiget_slice
    result.read(self._iprot)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 3816, in read
    fastbinary.decode_binary(self, iprot.trans, (self.__class__, self.thrift_spec))
OverflowError: list size out of the sanity limit (10000 items max)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19153
Committed: http://github.org/Juniper/contrail-controller/commit/2a9da59aa5887f8841a2f71c3a6e6b18a411cbd4
Submitter: Zuul
Branch: master

commit 2a9da59aa5887f8841a2f71c3a6e6b18a411cbd4
Author: Édouard Thuleau <email address hidden>
Date: Thu Apr 7 19:48:48 2016 +0200

[config] Remove all pycassa methods 'get' and 'multiget'

As pycassa methods 'get' and 'multiget' cannot fetch more than 10k
columns, we have to use 'xget' that returns a generator which pages
over the columns automatically to address scale issues.

Change-Id: I20675b5220b470b789bb14a949c56ea5340f216a
Related-bug: #1538991
Closes-bug: #1567566