commit 0a88916911e2b02055a2a707bda026c975f4472c
Author: Dan Smith <email address hidden>
Date: Thu Aug 16 13:52:01 2018 -0700
Batch results per cell when doing cross-cell listing
This extends the multi_cell_list module with batching support to avoid
querying N*$limit total results when listing resources across cells.
Instead, if our total limit is over a given threshold, we should query
smaller batches in the per-cell thread until we reach the total limit
or are stopped because the sort feeder has found enough across all cells
to satisfy the requirements. In many cases, this can drop the total number
of results we load and process from N*$limit to (best case) $limit+$batch
or (usual case) $limit+(N*$batch).
Since we return a generator from our scatter-gather function, this should
mean we basically finish the scatter immediately after the first batch query
to each cell database, keeping the threads alive until they produce all the
results possible from their cell, or are terminated in the generator loop
by the master loop hitting the total_limit condition. As a result, the
checking over results that we do immediately after the scatter finishes
will no longer do anything since we start running the query code for the
first time as heapq.merge() starts hitting the generators. So, this brings
a query_wrapper() specific to the multi_cell_list code which can mimic the
timeout and error handling abilities of scatter_gather_cells, but inline
as we're processing so that we don't interrupt the merge sort for a
failure.
Reviewed: https:/ /review. openstack. org/592698 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=0a88916911e 2b02055a2a707bd a026c975f4472c
Committed: https:/
Submitter: Zuul
Branch: master
commit 0a88916911e2b02 055a2a707bda026 c975f4472c
Author: Dan Smith <email address hidden>
Date: Thu Aug 16 13:52:01 2018 -0700
Batch results per cell when doing cross-cell listing
This extends the multi_cell_list module with batching support to avoid
querying N*$limit total results when listing resources across cells.
Instead, if our total limit is over a given threshold, we should query
smaller batches in the per-cell thread until we reach the total limit
or are stopped because the sort feeder has found enough across all cells
to satisfy the requirements. In many cases, this can drop the total number
of results we load and process from N*$limit to (best case) $limit+$batch
or (usual case) $limit+(N*$batch).
Since we return a generator from our scatter-gather function, this should gather_ cells, but inline
mean we basically finish the scatter immediately after the first batch query
to each cell database, keeping the threads alive until they produce all the
results possible from their cell, or are terminated in the generator loop
by the master loop hitting the total_limit condition. As a result, the
checking over results that we do immediately after the scatter finishes
will no longer do anything since we start running the query code for the
first time as heapq.merge() starts hitting the generators. So, this brings
a query_wrapper() specific to the multi_cell_list code which can mimic the
timeout and error handling abilities of scatter_
as we're processing so that we don't interrupt the merge sort for a
failure.
Related-Bug: #1787977 9bd735104d03d4d eec988d35a1
Change-Id: Iaa4759822e70b3