Implement migration between backends

Bug #814359 reported by Tristan Seligmann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Entropy Object Store
In Progress
Medium
Unassigned

Bug Description

[See also for more implementation details: bug #814366, bug #815495]

When a new backend is configured, any objects that are created subsequent to configuring the new backend will be uploaded to that backend via the upload scheduler, as normal. However, any objects created prior to this will not be migrated replicated to the new backend. There should be a way to make this happen en masse, and as a resumable process.

Currently, all objects are stored in the local content store, so the local store can always be used as the "source" of the migration. However, the migration API should allow for migrations using any pair of source and destination stores, as we are moving away from having the local store as a hardcoded-always-there backend.

The plan is to add a migrateTo method to IContentStore; to initiate a migration, call sourceStore.migrateTo(destinationStore). This would be required to be implemented such that once the migration completes, all objects stored in the sourceStore at the time of the migrateTo() call (and only those objects) will have been uploaded to destinationStore. The state of the migration would be tracked by an IMigration powerup; this would hold any state and behaviour specific to the IContentStore's implementation of migrateTo, as well as some generic methods for querying the status of the migration, pausing it, resuming it, cancelling it, and so on.

The state of the migration would not be replicated in any way beyond the local Entropy node; it is assumed that starting the migration from scratch on another node will be an acceptable option for recovery in the event of a catastrophic failure of the node that was managing the original migration attempt. This would necessarily involve migrating some objects that have already been uploaded to the new backend by the upload scheduler, but hopefully this will be a relatively small fraction of objects by comparison, and content stores are supposed to be idempotent with respect to the creation of objects, so the consequences should merely be a minor increase in the total cost (network/CPU/time) of the migration as a whole.

Related branches

description: updated
Changed in entropy-store:
status: New → In Progress
importance: Undecided → Medium
description: updated
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.