| 2006-06-07 12:12:39 |
Andrew Bennetts |
bug |
|
|
added bug |
| 2006-06-12 08:56:26 |
David Allouche |
launchpad-bazaar: status |
Unconfirmed |
Confirmed |
|
| 2006-06-12 08:56:26 |
David Allouche |
launchpad-bazaar: importance |
Untriaged |
High |
|
| 2006-06-12 08:56:26 |
David Allouche |
launchpad-bazaar: statusexplanation |
|
|
|
| 2006-06-16 13:57:39 |
Andrew Bennetts |
launchpad-bazaar: status |
Confirmed |
In Progress |
|
| 2006-06-16 13:57:39 |
Andrew Bennetts |
launchpad-bazaar: statusexplanation |
|
There's a workaround now in place that hopefully makes this much better. Let's see how it goes. |
|
| 2006-06-16 13:57:47 |
Andrew Bennetts |
launchpad-bazaar: assignee |
|
spiv |
|
| 2006-06-16 13:57:47 |
Andrew Bennetts |
launchpad-bazaar: statusexplanation |
There's a workaround now in place that hopefully makes this much better. Let's see how it goes. |
|
|
| 2006-06-22 12:26:06 |
Andrew Bennetts |
description |
SFTP hosted branches are stored locally, so mirroring them to HTTP should be very fast. Currently they're treated like external branches, and so there's generally many hours between pushing to SFTP and having it published over HTTP.
We should reduce this latency to as close as nothing as possible. To start with, perhaps we should poll all hosted branches everytime the branch puller cronscript runs?
The longer term fix would be to have the SFTP server inform the branch puller, probably by an XML-RPC call that updates the database, that a branch has been updated, so the puller can know which branches need attention. |
SFTP hosted branches should be mirrored http://bazaar.launchpad.net/ as close to instantly as practical.
We control the SFTP server, so we know when they are updated. Also hosted branches are stored locally, so mirroring them to HTTP should be very fast.
There is hack in place to mirror them all every time the branch puller cron script runs, this has brought the latency down to minutes (rather than a whole day!). This latency is acceptable (but not great), but this solution isn't scalable. The vast majority of hosted branches won't change between cron script invocations, but the script still has to examine each one of them. As more and more hosted branches are added (and as those branches grow more revisions), this script will become slower and slower, disproportionately to the amount of new branches and revisions that need to be mirrored.
To fix the scalability, the SFTP server should inform the branch puller, probably by an XML-RPC call that updates the database, that a branch has been updated, so the puller can know which branches need attention. This would have no impact on the latency (except that the cron script would be a bit faster because it doesn't need to examine unchanged hosted branches).
Also, it would be nice to further reduce the latency to almost nothing by having the SFTP server immediately kick off a branch mirroring attempt for a branch, rather than updating the database and waiting for the next run of the cronscript. Possibly this should be a topic of a new bug.
|
|
| 2006-06-22 12:26:06 |
Andrew Bennetts |
title |
Mirroring sftp hosted branches takes much too long |
Efficiently mirroring sftp hosted branches with minimal latency |
|
| 2006-08-29 15:42:57 |
David Allouche |
description |
SFTP hosted branches should be mirrored http://bazaar.launchpad.net/ as close to instantly as practical.
We control the SFTP server, so we know when they are updated. Also hosted branches are stored locally, so mirroring them to HTTP should be very fast.
There is hack in place to mirror them all every time the branch puller cron script runs, this has brought the latency down to minutes (rather than a whole day!). This latency is acceptable (but not great), but this solution isn't scalable. The vast majority of hosted branches won't change between cron script invocations, but the script still has to examine each one of them. As more and more hosted branches are added (and as those branches grow more revisions), this script will become slower and slower, disproportionately to the amount of new branches and revisions that need to be mirrored.
To fix the scalability, the SFTP server should inform the branch puller, probably by an XML-RPC call that updates the database, that a branch has been updated, so the puller can know which branches need attention. This would have no impact on the latency (except that the cron script would be a bit faster because it doesn't need to examine unchanged hosted branches).
Also, it would be nice to further reduce the latency to almost nothing by having the SFTP server immediately kick off a branch mirroring attempt for a branch, rather than updating the database and waiting for the next run of the cronscript. Possibly this should be a topic of a new bug.
|
SFTP hosted branches should be mirrored http://bazaar.launchpad.net/ as close to instantly as practical.
We control the SFTP server, so we know when they are updated. Also hosted branches are stored locally, so mirroring them to HTTP should be very fast.
There is hack in place to mirror them all every time the branch puller cron script runs, this has brought the latency down to minutes (rather than a whole day!). This latency is acceptable (but not great), but this solution isn't scalable. The vast majority of hosted branches won't change between cron script invocations, but the script still has to examine each one of them. As more and more hosted branches are added (and as those branches grow more revisions), this script will become slower and slower, disproportionately to the amount of new branches and revisions that need to be mirrored.
To fix the scalability, the SFTP server should inform the branch puller, probably by an XML-RPC call that updates the database, that a branch has been updated, so the puller can know which branches need attention. This would have no impact on the latency (except that the cron script would be a bit faster because it doesn't need to examine unchanged hosted branches).
Also, it would be nice to further reduce the latency to almost nothing by having the SFTP server immediately kick off a branch mirroring attempt for a branch, rather than updating the database and waiting for the next run of the cronscript. Possibly this should be a topic of a new bug.
Some discussion on how to implement that can be found in bug 49989. |
|
| 2007-02-08 06:50:15 |
Jonathan Lange |
launchpad-bazaar: importance |
High |
Critical |
|
| 2007-02-08 06:50:15 |
Jonathan Lange |
launchpad-bazaar: assignee |
spiv |
jml-mumak |
|
| 2007-02-08 06:50:15 |
Jonathan Lange |
launchpad-bazaar: statusexplanation |
|
This work is well under way. As Andrew has mentioned, the fix for this should fix 49989. The fix for this bug also implements the majority of https://launchpad.canonical.com/FasterBranchPullAndScan.
Updating to critical to be consistent with the other supermirror latency bugs. |
|
| 2007-02-15 11:59:10 |
Jonathan Lange |
launchpad-bazaar: status |
In Progress |
Fix Committed |
|
| 2007-02-15 11:59:10 |
Jonathan Lange |
launchpad-bazaar: statusexplanation |
This work is well under way. As Andrew has mentioned, the fix for this should fix 49989. The fix for this bug also implements the majority of https://launchpad.canonical.com/FasterBranchPullAndScan.
Updating to critical to be consistent with the other supermirror latency bugs. |
The fix should make the script a bit faster. Although this will have no _direct_ impact on latency, it will allow the cron script to be run much more frequently. Recommend that this be landed on production as soon as possible. |
|
| 2007-02-22 22:12:46 |
Jonathan Lange |
launchpad-bazaar: status |
Fix Committed |
Fix Released |
|
| 2007-02-22 22:12:46 |
Jonathan Lange |
launchpad-bazaar: statusexplanation |
The fix should make the script a bit faster. Although this will have no _direct_ impact on latency, it will allow the cron script to be run much more frequently. Recommend that this be landed on production as soon as possible. |
<jml> ddaa: and the fix for bug 48813 has landed?
<ddaa> jml: yes, yesterday |
|
