Threads provide poor performances improvement in Carbonara

[gordon chung]

threadpoolexecutor doesn't seem to do much. in fact it's a little bit worse when you have more than 1 thread.

below, the thread value, is the time it takes for _add_measures method[1] to complete. threads_total is just the cumulative time of all the threads working on the metric. it's not necessary total time since the threads may be done concurrently. in single thread case, it is (close) to total time.

[1] https://github.com/openstack/gnocchi/blob/master/gnocchi/storage/_carbonara.py#L204-L218

whether you have 24 threads:

2016-03-17 15:50:34.515 21193 ERROR gnocchi.storage._carbonara [-] thread: 2.462906853, threads_total: 2.462906853
2016-03-17 15:50:34.756 21193 ERROR gnocchi.storage._carbonara [-] thread: 2.702634491, threads_total: 5.165541344
2016-03-17 15:50:34.761 21193 ERROR gnocchi.storage._carbonara [-] thread: 2.704404852, threads_total: 7.86994619601
2016-03-17 15:50:34.762 21193 ERROR gnocchi.storage._carbonara [-] thread: 2.700864218, threads_total: 10.570810414
2016-03-17 15:50:34.883 21193 ERROR gnocchi.storage._carbonara [-] thread: 2.82034948, threads_total: 13.391159894
2016-03-17 15:50:35.305 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.129621745, threads_total: 16.520781639
2016-03-17 15:50:35.350 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.22499819403, threads_total: 19.745779833
2016-03-17 15:50:35.760 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.26446543098, threads_total: 26.522243694
2016-03-17 15:50:35.682 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.51199842998, threads_total: 23.257778263
2016-03-17 15:50:35.772 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.648191432, threads_total: 30.170435126
2016-03-17 15:50:36.148 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.95038992001, threads_total: 34.120825046
2016-03-17 15:50:36.197 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.91207420701, threads_total: 38.032899253
2016-03-17 15:50:36.206 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.64441813499, threads_total: 41.677317388
2016-03-17 15:50:36.211 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.991378081, threads_total: 45.668695469
2016-03-17 15:50:36.230 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.30182577399, threads_total: 48.970521243
2016-03-17 15:50:36.275 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.95342429099, threads_total: 52.923945534
2016-03-17 15:50:36.277 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.36573138801, threads_total: 56.289676922
2016-03-17 15:50:36.281 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.310435764, threads_total: 59.600112686
2016-03-17 15:50:36.353 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.92976355003, threads_total: 63.529876236
2016-03-17 15:50:36.418 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.89613281901, threads_total: 67.426009055
2016-03-17 15:50:36.425 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.233728864, threads_total: 70.659737919
2016-03-17 15:50:36.467 21193 ERROR gnocchi.storage._carbonara [-] thread: 3.40084429499, threads_total: 74.060582214
2016-03-17 15:50:36.494 21193 ERROR gnocchi.storage._carbonara [-] thread: 4.36836461799, threads_total: 78.428946832
2016-03-17 15:50:36.634 21193 ERROR gnocchi.storage._carbonara [-] thread: 4.51225105801, threads_total: 82.94119789

8 threads:

2016-03-17 15:54:32.570 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.634638646996, threads_total: 0.634638646996
2016-03-17 15:54:32.693 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.756876447005, threads_total: 1.391515094
2016-03-17 15:54:32.901 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.962289410993, threads_total: 2.35380450499
2016-03-17 15:54:33.306 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.37364586999, threads_total: 3.72745037498
2016-03-17 15:54:33.759 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.80439795699, threads_total: 5.53184833197
2016-03-17 15:54:33.791 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.83882889501, threads_total: 7.37067722698
2016-03-17 15:54:33.884 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.30180018701, threads_total: 8.67247741399
2016-03-17 15:54:33.939 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.01478598401, threads_total: 9.68726339799
2016-03-17 15:54:33.988 32145 ERROR gnocchi.storage._carbonara [-] thread: 2.00655171898, threads_total: 11.693815117
2016-03-17 15:54:34.020 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.31234327401, threads_total: 13.006158391
2016-03-17 15:54:34.383 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.036197126, threads_total: 14.042355517
2016-03-17 15:54:34.937 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.17471699, threads_total: 15.217072507
2016-03-17 15:54:35.266 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.31295781498, threads_total: 16.530030322
2016-03-17 15:54:35.376 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.56815874702, threads_total: 18.098189069
2016-03-17 15:54:35.392 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.495416919, threads_total: 19.593605988
2016-03-17 15:54:35.469 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.47162978697, threads_total: 21.065235775
2016-03-17 15:54:35.520 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.48434645901, threads_total: 22.549582234
2016-03-17 15:54:35.652 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.26822512699, threads_total: 23.817807361
2016-03-17 15:54:36.116 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.842649034981, threads_total: 24.6604563959
2016-03-17 15:54:36.160 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.766899356997, threads_total: 25.4273557529
2016-03-17 15:54:36.161 32145 ERROR gnocchi.storage._carbonara [-] thread: 1.21299281198, threads_total: 26.6403485649
2016-03-17 15:54:36.189 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.796621607995, threads_total: 27.4369701729
2016-03-17 15:54:36.298 32145 ERROR gnocchi.storage._carbonara [-] thread: 0.789528068999, threads_total: 28.2264982419
2016-03-17 15:54:36.400 32145 ERROR gnocchi.storage._carbonara [-] thread: 4.43755432498, threads_total: 32.6640525669

or 1 thread:

2016-03-17 16:06:26.260 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.103176662989, threads_total: 0.103176662989
2016-03-17 16:06:26.367 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.105849467014, threads_total: 0.209026130004
2016-03-17 16:06:26.472 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.104437161004, threads_total: 0.313463291008
2016-03-17 16:06:26.573 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.100709477992, threads_total: 0.414172769
2016-03-17 16:06:26.737 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.163463389006, threads_total: 0.577636158006
2016-03-17 16:06:26.843 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.105480414, threads_total: 0.683116572007
2016-03-17 16:06:27.117 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.273747911997, threads_total: 0.956864484004
2016-03-17 16:06:27.234 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.116017488996, threads_total: 1.072881973
2016-03-17 16:06:27.339 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.103926073993, threads_total: 1.17680804699
2016-03-17 16:06:27.436 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.0964345749817, threads_total: 1.27324262197
2016-03-17 16:06:27.605 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.168536409008, threads_total: 1.44177903098
2016-03-17 16:06:27.703 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.095962641004, threads_total: 1.53774167199
2016-03-17 16:06:27.803 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.0984709449986, threads_total: 1.63621261698
2016-03-17 16:06:27.923 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.118593288993, threads_total: 1.75480590598
2016-03-17 16:06:28.031 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.106659351994, threads_total: 1.86146525797
2016-03-17 16:06:28.127 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.0949174579873, threads_total: 1.95638271596
2016-03-17 16:06:28.232 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.103710158, threads_total: 2.06009287396
2016-03-17 16:06:28.420 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.187739057001, threads_total: 2.24783193096
2016-03-17 16:06:28.549 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.128680551017, threads_total: 2.37651248198
2016-03-17 16:06:28.662 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.111980563001, threads_total: 2.48849304498
2016-03-17 16:06:28.783 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.120761684986, threads_total: 2.60925472996
2016-03-17 16:06:28.914 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.130160519999, threads_total: 2.73941524996
2016-03-17 16:06:29.032 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.117343777005, threads_total: 2.85675902697
2016-03-17 16:06:29.147 5188 ERROR gnocchi.storage._carbonara [-] thread: 0.113576553995, threads_total: 2.97033558096

[Julien Danjou (jdanjou)]

Can you explain the numbers? This is the number of total threads running at the same time vs the number of thread already run? Or something else?

[gordon chung (chungg)]

sure, i'll edit the description

[Julien Danjou (jdanjou)]

I updated the title. Python threading sucks, news at 11.

Now I don't have any other solution than giving up on CPython, what's yours?

[gordon chung (chungg)]

i'm googling. :)

[gordon chung (chungg)]

i don't have a solution for this. maybe we should default threading to 1.

i haven't confirmed if threading benefits ceph, it hurts performance in file backend. also haven't tried with swift backend because i can't track logs properly because of all the random swiftclient error logs

[Julien Danjou (jdanjou)]

Using the file backend I did some tests.

With 1 thread:

Computed new metric 14ac5217-f879-43b7-9c85-ac224ab7561d with 50000 new measures in 104.19 seconds (11517 points/s)

with 12 threads (12 CPU):

Computed new metric a22ca938-9d0f-4c08-9b86-6afecb6d0a79 with 50000 new measures in 121.94 seconds (9840 points/s)

So it seems you're right. Amazing Python.

Ceph and Swift shall benefit more of parallelism since the I/O latency is going to higher, so serializing the read/write calls would increase the time to process metrics.

I'll try with process instead of threads if that's doable.

[gordon chung (chungg)]

not sure if it's worth implementing process instead of threads. i do recall it being more heavyweight than threads and meant more for longer running actions (not millisecond stuff we're doing). it also requires you to make everything serialisable so it may be non-trivial fix.

the only real alternative i found was using cython to turn off gil, but it's also a pretty big change.

http://docs.cython.org/src/userguide/parallelism.html

[gordon chung (chungg)]

this is actually worse with ceph too... and i have 70-80% IO. we should disable this by default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/330979

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/330979
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=3156b9d383762a2b4297f3416ccd5205e22c16ef
Submitter: Jenkins
Branch: master

commit 3156b9d383762a2b4297f3416ccd5205e22c16ef
Author: Julien Danjou <email address hidden>
Date: Fri Jun 17 11:20:08 2016 +0200

    _carbonara: set default aggregation_workers_number to 1

    Change-Id: Iaa2ca23611d37cf96262104829dc96cb9ef6b0bd
    Closes-Bug: #1558791

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/gnocchi 2.2.0 release.