Computed new metric 14ac5217-f879-43b7-9c85-ac224ab7561d with 50000 new measures in 104.19 seconds (11517 points/s)
with 12 threads (12 CPU):
Computed new metric a22ca938-9d0f-4c08-9b86-6afecb6d0a79 with 50000 new measures in 121.94 seconds (9840 points/s)
So it seems you're right. Amazing Python.
Ceph and Swift shall benefit more of parallelism since the I/O latency is going to higher, so serializing the read/write calls would increase the time to process metrics.
I'll try with process instead of threads if that's doable.
Using the file backend I did some tests.
With 1 thread:
Computed new metric 14ac5217- f879-43b7- 9c85-ac224ab756 1d with 50000 new measures in 104.19 seconds (11517 points/s)
with 12 threads (12 CPU):
Computed new metric a22ca938- 9d0f-4c08- 9b86-6afecb6d0a 79 with 50000 new measures in 121.94 seconds (9840 points/s)
So it seems you're right. Amazing Python.
Ceph and Swift shall benefit more of parallelism since the I/O latency is going to higher, so serializing the read/write calls would increase the time to process metrics.
I'll try with process instead of threads if that's doable.