librarian-feed-swift needs to upload faster (parallel) at times to keep up
Bug #1599001 reported by
Michael Foley
This bug affects 2 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Colin Watson |
Bug Description
When librarian-
Related branches
~cjwatson/launchpad:parallel-feed-swift
Merged
into
launchpad:master
- Ioana Lasc (community): Approve
-
Diff: 168 lines (+90/-4)3 files modifiedcronscripts/librarian-feed-swift.py (+35/-3)
lib/lp/services/librarianserver/swift.py (+14/-1)
lib/lp/services/librarianserver/tests/test_swift.py (+41/-0)
tags: | added: librarian |
Changed in launchpad: | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Colin Watson (cjwatson) |
Changed in launchpad: | |
status: | In Progress → Fix Released |
To post a comment you must log in.
The simplest approach is to add --instance-id and --num-instances arguments, and have feed swift skip files unless (LibraryFileCon tent.id % num_instances= =instance_ id). This and updating the script locking to allow parallel runs is the least invasive approach, allowing us to easily schedule n tasks in parallel.
IIRC feed-swift will eventually disappear in favour of direct uploads to Swift, removing a SPOF.