Fail to fetch ressource lists if there is more than 10k elements

[Édouard Thuleau]

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 537, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/gen/vnc_api_server_gen.py", line 13387, in access_control_lists_http_get
    db_conn.dbe_list('access-control-list', parent_uuids, back_ref_uuids, obj_uuids, count, filters)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 1478, in dbe_list
    count=count, filters=filters)
  File "/usr/local/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 546, in list
    return method(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/vnc_api/gen/vnc_cassandra_client_gen.py", line 9958, in _cassandra_access_control_list_list
    cols = obj_fq_name_cf.get('access_control_list', column_count = 10000000)
  File "/usr/local/lib/python2.7/dist-packages/cfgm_common/vnc_cassandra.py", line 109, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/columnfamily.py", line 660, in get
    read_consistency_level or self.read_consistency_level)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 577, in execute
    return getattr(conn, f)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/pool.py", line 127, in new_f
    result = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 580, in get_slice
    return self.recv_get_slice()
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 601, in recv_get_slice
    result.read(self._iprot)
  File "/usr/local/lib/python2.7/dist-packages/pycassa/cassandra/Cassandra.py", line 3391, in read
    fastbinary.decode_binary(self, iprot.trans, (self.__class__, self.thrift_spec))
OverflowError: list size out of the sanity limit (10000 items max)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18486
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18486
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18486
Committed: http://github.org/Juniper/contrail-controller/commit/8de9c9a637804a107ea81275d59dc20a64c19901
Submitter: Zuul
Branch: master

commit 8de9c9a637804a107ea81275d59dc20a64c19901
Author: Édouard Thuleau <email address hidden>
Date: Wed Mar 16 17:51:24 2016 +0100

Use pycassa xget() instead of get()

When listing all resources of a type without any filters,
the get() method is used to fetch data from cassandra and can fail
if the amount of resources exceed 10k due to a thrift limitation.
Use of xget() method resolves that.

Change-Id: I4ba804ed3f400118eb5c28b1d5be0cf7a7a6cedc
Closes-Bug: #1538991

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19392
Submitter: Jean-Philippe Braun (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19392
Committed: http://github.org/Juniper/contrail-generateDS/commit/0cc60117acf21b0710241fbbbb03fb8374e7c8f6
Submitter: Zuul
Branch: R2.21.x

commit 0cc60117acf21b0710241fbbbb03fb8374e7c8f6
Author: Jean-Philippe Braun <email address hidden>
Date: Mon Apr 18 16:05:47 2016 +0200

Use pycassa xget() instead of get()

When listing all resources of a type without any filters,
the get() method is used to fetch data from cassandra and can fail
if the amount of resources exceed 10k due to a thrift limitation.
Use of xget() method resolves that.

Change-Id: I375ada4decd97ee4f1aeba35b74acd8867b22e5b
Closes-Bug: #1538991

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20322
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19153
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20322
Committed: http://github.org/Juniper/contrail-controller/commit/d8efaf3990c546d2d13db6cd908ae015ac5d4347
Submitter: Zuul
Branch: R2.21.x

commit d8efaf3990c546d2d13db6cd908ae015ac5d4347
Author: Megh Bhatt <email address hidden>
Date: Tue May 17 11:47:46 2016 -0700

Add missing column_count to xget in FakeCF so that CI unit-tests pass

Change-Id: I695f7b0c27bb90e4096489f9fe5f1fe0015ee5a9
Closes-Bug: #1538991

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19153
Committed: http://github.org/Juniper/contrail-controller/commit/2a9da59aa5887f8841a2f71c3a6e6b18a411cbd4
Submitter: Zuul
Branch: master

commit 2a9da59aa5887f8841a2f71c3a6e6b18a411cbd4
Author: Édouard Thuleau <email address hidden>
Date: Thu Apr 7 19:48:48 2016 +0200

[config] Remove all pycassa methods 'get' and 'multiget'

As pycassa methods 'get' and 'multiget' cannot fetch more than 10k
columns, we have to use 'xget' that returns a generator which pages
over the columns automatically to address scale issues.

Change-Id: I20675b5220b470b789bb14a949c56ea5340f216a
Related-bug: #1538991
Closes-bug: #1567566