Should allow more room for LP's clock to go backwards

[James Westby]

Hi,

Currently we give LP 10 minutes leeway for clock skew (against itself, not against us).

icommon.create_import_jobs() does:

    if last_known_published is not None:
        last_known_published = \
                last_known_published - datetime.timedelta(0, 600)
        last_published_call = last_known_published.isoformat()

We should allow more than this.

This will increase the load against LP a little, but shouldn't be too much, as it will just
be serialising a few more rows per query, though this will actually require more round
trips if we fetch more pages.

Currently it will also increase the load on the importer, as we will keep seeing the same
publications repeatedly. We should however rely on the fact that LP should never publish
the same name/version in a series/pocket twice. It will be very rare that this happens, if
ever. We could perhaps be more robust by checking source publication ids, rather than
name/version pairs, however I don't know if these will be reused as well in those rare cases.

Thanks,

James

[James Westby (james-w)]

The reason for this is that we missed the publication of a new puppet package, and this is
the only hole I know for how that could have happened.

Thanks,

James

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/25/2010 11:17 AM, James Westby wrote:
> The reason for this is that we missed the publication of a new puppet package, and this is
> the only hole I know for how that could have happened.
>
> Thanks,
>
> James
>

Do you know what range we could have sent to have seen it? (For example,
if we missed it by 2 days, then no level would be reasonable. If we
missed it by 11min, then maybe 30min is reasonable.)

Note that I think the ideal version would just use a work queue that the
scanner publishes. So when a new package is seen, it gets marked as
'needs to be imported' in a queue, and the importer removes stuff from
that only when the import has succeeded (or irrevocably failed).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkx1SYQACgkQJdeBCYSNAAMj2wCeOXdoKG4Xlf6hvlWU+1JBcYDZ
IeAAnRW77eAXLxtaiIJaxf9ljJXyKqp6
=x2iv
-----END PGP SIGNATURE-----

[James Westby (james-w)]

On Wed, 25 Aug 2010 11:49:08 -0500, John Arbash Meinel <email address hidden> wrote:
> Do you know what range we could have sent to have seen it? (For example,
> if we missed it by 2 days, then no level would be reasonable. If we
> missed it by 11min, then maybe 30min is reasonable.)

I don't know what we missed it by. I don't even know that this was the
cause. It is the only cause I know of right now, but it is more likely
to be a bug somewhere, I just don't know how to track down the bug after
the fact.

> Note that I think the ideal version would just use a work queue that the
> scanner publishes. So when a new package is seen, it gets marked as
> 'needs to be imported' in a queue, and the importer removes stuff from
> that only when the import has succeeded (or irrevocably failed).

Yes, that's what I want to do.

I have started making notes here on how that will work:

  https://dev.launchpad.net/Code/PackageImporter

Note that what you describe is exactly how it works now, except that we
have to poll to find out what happened. Under a perfectly valid set of
assumptions this should ensure that we get all packages, but I would
like to move to the more robust solution anyway.

Thanks,

James