The simplest approach is to add --instance-id and --num-instances arguments, and have feed swift skip files unless (LibraryFileContent.id % num_instances==instance_id). This and updating the script locking to allow parallel runs is the least invasive approach, allowing us to easily schedule n tasks in parallel.
IIRC feed-swift will eventually disappear in favour of direct uploads to Swift, removing a SPOF.
The simplest approach is to add --instance-id and --num-instances arguments, and have feed swift skip files unless (LibraryFileCon tent.id % num_instances= =instance_ id). This and updating the script locking to allow parallel runs is the least invasive approach, allowing us to easily schedule n tasks in parallel.
IIRC feed-swift will eventually disappear in favour of direct uploads to Swift, removing a SPOF.