Implement migration between backends
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Entropy Object Store |
In Progress
|
Medium
|
Unassigned |
Bug Description
[See also for more implementation details: bug #814366, bug #815495]
When a new backend is configured, any objects that are created subsequent to configuring the new backend will be uploaded to that backend via the upload scheduler, as normal. However, any objects created prior to this will not be migrated replicated to the new backend. There should be a way to make this happen en masse, and as a resumable process.
Currently, all objects are stored in the local content store, so the local store can always be used as the "source" of the migration. However, the migration API should allow for migrations using any pair of source and destination stores, as we are moving away from having the local store as a hardcoded-
The plan is to add a migrateTo method to IContentStore; to initiate a migration, call sourceStore.
The state of the migration would not be replicated in any way beyond the local Entropy node; it is assumed that starting the migration from scratch on another node will be an acceptable option for recovery in the event of a catastrophic failure of the node that was managing the original migration attempt. This would necessarily involve migrating some objects that have already been uploaded to the new backend by the upload scheduler, but hopefully this will be a relatively small fraction of objects by comparison, and content stores are supposed to be idempotent with respect to the creation of objects, so the consequences should merely be a minor increase in the total cost (network/CPU/time) of the migration as a whole.
Related branches
- Jonathan Jacobs: Approve
-
Diff: 694 lines (+503/-18)4 files modifiedentropy/ientropy.py (+68/-1)
entropy/s3.py (+2/-0)
entropy/store.py (+200/-4)
entropy/test/test_store.py (+233/-13)
description: | updated |
Changed in entropy-store: | |
status: | New → In Progress |
importance: | Undecided → Medium |
description: | updated |
description: | updated |
description: | updated |