[DM] ZEO large transaction optimization

[Andreas Jung]

2. Introduction

ZEO is essentially a "single request at a time server". This is motivated by the fact that almost all ZEO requests are highly IO bound and performing several IO operations in parallel does often not improve but even deteriorate the overall throughput (through IO thrashing).

There are, however, some especially long running requests that ZEO performs in a separate thread: especially packing, and "Version" operations ("Version abort/commit") as they affect a huge amount of objects and can take very long.

Transactions, too, can be large, affect a large number of objects and therefore take a considerable time. Colleagues occationally executed such transactions (catalog reindex, usually) and thereby caused severe interference with the cluster supervision (the cluster probe was not answered in a timely fashion, because the transaction vote took several minutes).

Therefore, I changed ZEO to execute the vote for large transactions (either a large number of affected objects or a large total size) in a separate thread. This allows other requests to be executed in parallel.

3. Feature

Perform the vote for sufficiently large transactions in a separate thread and thereby avoid ZEO client blocking for an excessive timespan.

4. Example Use Cases

Avoid catalog reindexing to cause the cluster to restart ZEO.

[Jim Fulton (jim-zope)]

This is interesting.

One of my upcoming projects is to revisit the single-threaded(-ish) ZEO server architecture. IOW, a major ZEO server rework is coming.

[Jim Fulton (jim-zope)]

This is a feature request, not a bug report.

See:
https://blueprints.launchpad.net/zodb/+spec/dm-large-transaction-opt