Cassandra ObjectCacheManager is O(keys_in_cache) per request

[Ankudinov Alexander]

We've made a test with more than 10000 ports and a lot of requests to API on R3.2.

During this test we've found, that contrail-api uses 100% cpu, performed profiling and found out, that most of the time is used by ObjectCacheManager.

The profiling graph, made with py-spy is attached.

The time is mostly used by creating a set from all cache keys and set operations on each request.

P.S. It's my first time contributing to TF, please tell me if I filled anything wrong.

[Ankudinov Alexander (xelez)]

I've made a patch here, but I can't submit it to gerrit until my CCLA is sorted out.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47922
Submitter: Ankudinov Alexander (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48057
Submitter: Ankudinov Alexander (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/48057
Committed: http://github.com/Juniper/contrail-controller/commit/ab30adcff0f8d4a482d8bde578f09d9708ee86ed
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit ab30adcff0f8d4a482d8bde578f09d9708ee86ed
Author: Alexander Ankudinov <email address hidden>
Date: Tue Nov 20 17:55:07 2018 +0500

Optimize ObjectCacheManager

Don't use sets when calculating cache hit

Test results on a database with around 16000 floating ips
and 10000 VMs (for original patch for R3.2).

Without patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m9.444s (+- 0.1 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m23.466s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 146.96 [#/sec] (mean)
Time per request: 6.805 [ms] (mean)

With patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m3.107s (+- 2 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m7.127s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 271.36 [#/sec] (mean)
Time per request: 3.685 [ms] (mean)

Closes-bug: 1804201
Change-Id: I3150787bfbc61646ecd7006040c649b83f36c1a7

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/47922
Submitter: Ankudinov Alexander (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48068
Submitter: Ankudinov Alexander (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48072
Submitter: Ankudinov Alexander (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/47922
Committed: http://github.com/Juniper/contrail-controller/commit/551cd29cedc0baffe7afa674c878b630db4a711a
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 551cd29cedc0baffe7afa674c878b630db4a711a
Author: Alexander Ankudinov <email address hidden>
Date: Tue Nov 20 17:55:07 2018 +0500

Optimize ObjectCacheManager

* don't use sets when calculating cache hit
* cleaner evicting from cache

Test results on a database with around 16000 floating ips
and 10000 VMs.

Without patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m9.444s (+- 0.1 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m23.466s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 146.96 [#/sec] (mean)
Time per request: 6.805 [ms] (mean)

With patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m3.107s (+- 2 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m7.127s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 271.36 [#/sec] (mean)
Time per request: 3.685 [ms] (mean)

Closes-bug: 1804201
Change-Id: I3150787bfbc61646ecd7006040c649b83f36c1a7

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/48068
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/48072
Committed: http://github.com/Juniper/contrail-controller/commit/c2502b8ed38037715d9f1eb8bf5f15184ef5f4d2
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit c2502b8ed38037715d9f1eb8bf5f15184ef5f4d2
Author: Alexander Ankudinov <email address hidden>
Date: Tue Nov 20 17:55:07 2018 +0500

Optimize ObjectCacheManager

Don't use sets when calculating cache hit

Test results on a database with around 16000 floating ips
and 10000 VMs (for original patch for R3.2).

Without patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m9.444s (+- 0.1 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m23.466s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 146.96 [#/sec] (mean)
Time per request: 6.805 [ms] (mean)

With patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m3.107s (+- 2 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m7.127s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 271.36 [#/sec] (mean)
Time per request: 3.685 [ms] (mean)

Closes-bug: 1804201
Change-Id: I3150787bfbc61646ecd7006040c649b83f36c1a7

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/48068
Committed: http://github.com/Juniper/contrail-controller/commit/18e198ff4a6c9aeb9a6dc08fc9f1aa5dd4e70ba3
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 18e198ff4a6c9aeb9a6dc08fc9f1aa5dd4e70ba3
Author: Alexander Ankudinov <email address hidden>
Date: Tue Nov 20 17:55:07 2018 +0500

Optimize ObjectCacheManager

Don't use sets when calculating cache hit

Test results on a database with around 16000 floating ips
and 10000 VMs (for original patch for R3.2).

Without patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m9.444s (+- 0.1 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m23.466s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 146.96 [#/sec] (mean)
Time per request: 6.805 [ms] (mean)

With patch:

$ time wget 'http://10.100.12.171:8082/virtual-machine-interfaces?count=False'
real 0m3.107s (+- 2 seconds)
$ time wget 'http://10.100.12.171:8082/floating-ips?count=False'
real 0m7.127s (+- 1 seconds)
$ ab -n3000 http://10.100.12.171:8082/floating-ip/c0e90af8-f10c-4202-ad01-357b0e369442
Requests per second: 271.36 [#/sec] (mean)
Time per request: 3.685 [ms] (mean)

Closes-bug: 1804201
Change-Id: I3150787bfbc61646ecd7006040c649b83f36c1a7