Use bileto instead of rsync for copy2distro script.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Bileto |
Triaged
|
Wishlist
|
Unassigned | ||
CI Train [cu2d] |
Won't Fix
|
Undecided
|
Unassigned |
Bug Description
Currently copy2distro script is in rough shape. It's the only part of the train that doesn't have test coverage, and it's using rsync in a somewhat archaic way, which made sense back in the spreadsheet days, but doesn't make sense now that we have bileto.
What I'd like to see is bileto grow a new table called "publications" which has the same fields as defined by the current PackageList class (all the info necessary to do a copyPackage from a silo to a distro archive).
Once that's in place, copy2distro script can be completely rewritten to stop rsyncing files of jenkins instance but instead query json from public bileto, and then call copyPackage that way.
This will simplify jenkins (which can then entirely drop rsyncd), eliminate the PackageList class entirely, and drop an enormous class of bugs that keep biting us, caused by copy2distro being python2 while the entire rest of the train is python3, but the unittests are only python3 so it doesn't catch python2 regressions in modules imported by copy2distro.
Changed in bileto: | |
status: | New → Triaged |
assignee: | nobody → Robert Bruce Park (robru) |
Changed in cupstream2distro: | |
importance: | Undecided → Medium |
Changed in bileto: | |
importance: | Undecided → Medium |
Changed in cupstream2distro: | |
status: | Triaged → Won't Fix |
Changed in cupstream2distro: | |
importance: | Medium → Undecided |
Changed in bileto: | |
importance: | Medium → Wishlist |
Changed in cupstream2distro: | |
assignee: | Robert Bruce Park (robru) → nobody |
Changed in bileto: | |
assignee: | Robert Bruce Park (robru) → nobody |
Just discussed this with Steve & Lukasz and made some decisions:
* The bileto API should have a GET parameter that specifies a minimum row id, so that bileto doesn't send old rows to snakefruit for publishing.
* Bileto should also only return records that are not known to be fully migrated, in the event that snakefruit dies and is redeployed and we lose the local state of the most recent published id.