ImportDriver should wait between ThreadedImporter spawns

[Vincent Ladeuil]

During the wheezy import we encountered a rare race condition while spawning imports.

I say 'rare' because we didn't encounter it during the 15,000 wheezy imports nor during imports covered by our log files (back to 2011-01-16).

The likely culprit is ImportDriver.run() which doesn't wait between spawns.

[Vincent Ladeuil (vila)]

Instead of "doesn't wait between spawns" I think it would be more accurate to say: "doesn't wait until an import has fully spawned before starting the next". What seems to have occur during the wheezy import is that two imports tried to lock the 'meta.db' database at the same time and this operation occurs before the 'import_package.py' script is spawned as a subprocess.

At least, that's my current theory.

[Vincent Ladeuil (vila)]

Relevant log excerpt :

2011-02-13 19:56:59,671 - __main__ - CRITICAL - Driver hit database is locked
2011-02-13 19:56:59,672 - __main__ - INFO - Driver stopping
2011-02-13 19:56:59,693 - __main__ - INFO - Waiting for driver to finish
2011-02-13 19:56:59,694 - __main__ - INFO - Finished

It would be nice to keep the traceback here so we don't have to guess too much if/when we encounter similar issues.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/15/2011 6:40 AM, Vincent Ladeuil wrote:
> Instead of "doesn't wait between spawns" I think it would be more
> accurate to say: "doesn't wait until an import has fully spawned before
> starting the next". What seems to have occur during the wheezy import is
> that two imports tried to lock the 'meta.db' database at the same time
> and this operation occurs before the 'import_package.py' script is
> spawned as a subprocess.
>
> At least, that's my current theory.
>

I think there is always a race condition with multiple processes sharing
a single sqlite meta.db. It just happens that writes are rare, and so
you rarely get contention. But it is entirely possible for 2 of them to
want to update the meta.db at the same time, while imports are processing.

I suppose it is possible that other places in the code already expected
this, and handle contention issues.

But sqlite is single-writer, and each package_import.py script will be
wanting to write to it.

Though over the minutes that an import runs, it will only need to write
for a few ms. So rare, but certainly possible.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1amxYACgkQJdeBCYSNAAPqIgCfbOCCoU0hUxhW12RFBtt6yQPw
pbYAoIEfbe1coe8G3URT1zFXr3Em/CNm
=BkLS
-----END PGP SIGNATURE-----

[Vincent Ladeuil (vila)]

The new driver has landed with a fix addressing the race I mentioned in comment #2.

If other races are encountered as jam suspects, we'll open a new bug.