Allow purging old transaction log entries

[John A Meinel]

We want to avoid having any data that must be maintained forever that is not going to be actively used.

One of the O(history) tables today is the transaction_log table. It should be possible to remove old entries, and if a client connects and tries to sync from that generation, we fall back to a 'full sync'.

Along these lines, when you remove the old generations, you need to move the doc_ids that are current somewhere so that you can do the full sync. (It breaks the current notion that each generation corresponds to a single document change.)

[John A Meinel (jameinel)]

At the basic concept, this should give correct results, and only lose the ability to do partial synchronization from the revisions that have been purged. (If you request a sync from before the history horizon, you just end up with a full sync, getting some documents that you already have.)

[John A Meinel (jameinel)]

Note that the big win here is if you have documents that change a lot, but not a lot of documents. As a limit case, imagine 1 document, changed 10000 times. The transaction log would be 10k long, but after purging, you only have 1 doc_id you need to store.
As the other limit case, 10k documents, only created (never modified). Purging the log will just move where the doc_ids are stored, not actually saving any space (but maybe saving some index meta-info space, etc.)

[John A Meinel (jameinel)]

If the expectation is that this data won't be accessed often, you could also do post-processing. For example, you could store the rollup as a compressed string of delimited doc_ids.