Use bileto instead of rsync for copy2distro script.

Bug #1508186 reported by Robert Bruce Park
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Bileto
Triaged
Wishlist
Unassigned
CI Train [cu2d]
Won't Fix
Undecided
Unassigned

Bug Description

Currently copy2distro script is in rough shape. It's the only part of the train that doesn't have test coverage, and it's using rsync in a somewhat archaic way, which made sense back in the spreadsheet days, but doesn't make sense now that we have bileto.

What I'd like to see is bileto grow a new table called "publications" which has the same fields as defined by the current PackageList class (all the info necessary to do a copyPackage from a silo to a distro archive).

Once that's in place, copy2distro script can be completely rewritten to stop rsyncing files of jenkins instance but instead query json from public bileto, and then call copyPackage that way.

This will simplify jenkins (which can then entirely drop rsyncd), eliminate the PackageList class entirely, and drop an enormous class of bugs that keep biting us, caused by copy2distro being python2 while the entire rest of the train is python3, but the unittests are only python3 so it doesn't catch python2 regressions in modules imported by copy2distro.

Changed in bileto:
status: New → Triaged
assignee: nobody → Robert Bruce Park (robru)
Changed in cupstream2distro:
importance: Undecided → Medium
Changed in bileto:
importance: Undecided → Medium
Revision history for this message
Robert Bruce Park (robru) wrote :

Just discussed this with Steve & Lukasz and made some decisions:

* The bileto API should have a GET parameter that specifies a minimum row id, so that bileto doesn't send old rows to snakefruit for publishing.

* Bileto should also only return records that are not known to be fully migrated, in the event that snakefruit dies and is redeployed and we lose the local state of the most recent published id.

Changed in cupstream2distro:
status: Triaged → Won't Fix
Changed in cupstream2distro:
importance: Medium → Undecided
Changed in bileto:
importance: Medium → Wishlist
Changed in cupstream2distro:
assignee: Robert Bruce Park (robru) → nobody
Changed in bileto:
assignee: Robert Bruce Park (robru) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.