Package Auto Sync seems to get ahead of +localdiff calculation

[Dave Walker]

For example:

          Sid | Oneiric
aafigure 0.5-1 | 0.5-2

Launchpad hasn't yet noticed that Sid contains 0.5-2, at time of writing, https://launchpad.net/debian/sid/+source/aafigure/0.5-2 returns 404. However, it is clearly in Sid, as it's been Auto Sync'd to Oneiric.

Additionally, If we (Ubuntu), are before Debian Import Freeze - i'm not sure it matters to show there is a localdiff between Ubuntu and Debian, as it should be automatically resolved within a small period of time.

Thanks.

[Julian Edwards (julian-edwards)]

I suspect someone synced from Debian before our local mirror was updated and imported into LP. We currently import Debian into Launchpad on this cron schedule:

05 3,9,15,21

i.e. 4 times a day. Does Debian update its repo more often?

>Additionally, If we (Ubuntu), are before Debian Import Freeze - i'm not sure it matters
>to show there is a localdiff between Ubuntu and Debian, as it should be automatically
>resolved within a small period of time.

It'll catch up quickly though, it's a very small corner case.

[Colin Watson (cjwatson)]

Debian updates four times a day at the moment, but I suspect that you sometimes end up just missing it depending on how long its publisher equivalent happens to take. Assuming the importer is fairly quick to do nothing, then you could just try updating every hour or something like that?

[Julian Edwards (julian-edwards)]

Here's some figures:

Debian publishes at:
52 1,7,13,19

The LP mirroring happens at:
52 2,8,14,20
and seems to take no more than a minute. (It mirrors from ftp.uk.debian.org)

As mentioned above, we run the LP import at:
05 3,9,15,21

So we could make the mirroring happen just before the import, or combine the 2 scripts. In any case, running the mirroring later won't have any effect on when LP gets updated but will allow more time for the Debian mirror to update itself.

[Julian Edwards (julian-edwards)]

Oh and the importer takes almost 30 minutes, so we can't run it more often without impacting LP performance generally (albeit a small impact, but nonetheless ...)

[Colin Watson (cjwatson)]

Those figures suggest that a good strategy would be to run the
relatively cheap mirroring part much more frequently (e.g. hourly) and
to then kick off the expensive importer part if and only if the mirror
timestamps indicate that something actually changed?

[Julian Edwards (julian-edwards)]

On Monday 22 August 2011 13:50:30 you wrote:
> Those figures suggest that a good strategy would be to run the
> relatively cheap mirroring part much more frequently (e.g. hourly) and
> to then kick off the expensive importer part if and only if the mirror
> timestamps indicate that something actually changed?

Agreed. I am looking at trying to merge the two separate cron jobs.

[Julian Edwards (julian-edwards)]

For now I'm pushing the import back a little, and closing the gap between mirroring and importing by making mirroring run later. This will hopefully give the best chance of getting all the packages imported in a timely fashion, without the extra work of messing with scripts.

[Julian Edwards (julian-edwards)]

Re-opening as it's still a bit of a problem.

[Colin Watson (cjwatson)]

I don't know if this is anomalous as I haven't been observing it regularly, but the last sync to ftp.uk.debian.org completed at 15:47 UTC. Assuming that iron's crontab times are also in UTC, this means that the current import timing is very nearly pessimal, and moving it an hour later would probably decrease latency from about six hours to about one hour.

I'll keep an eye on this over the next couple of days to see if this is normal.

[Colin Watson (cjwatson)]

Incidentally, the subject of this bug can no longer be true as such, as autosyncs now copy publication records so they require the Debian import to be up to date first. Still, that just means that import delays tend to delay autosyncs too. Compare bug 812597.

[Colin Watson (cjwatson)]

ftp.uk.debian.org sync completion times since comment 9:

  21:42
  03:15
  09:02

[Colin Watson (cjwatson)]

And the latest is 15:43. I don't think the contents of these mirror pushes have been particularly atypical, so this seems to be a pretty clear indication that the cron job is mistimed, insofar as it's a good idea for it to be a six-hourly cron job at all.

[Colin Watson (cjwatson)]

Steve McInerney has kindly moved the cron jobs one hour later for me. Hopefully this should help.