need less verbose logging, more efficient prerequisite processing

[siznax]

currently, always get verbose output, eg

    MANIFEST exists: /3/incoming/newscrawl/NEWS-20100901201302653-00001-00019-ia360914/MANIFEST
    ... (repeated many times) ...
    LAUNCH file exists: /3/incoming/newscrawl/NEWS-20100901201302653-00001-00019-ia360914/LAUNCH
    ... (repeated many times) ...
    TOMBSTONE exits: /3/incoming/newscrawl/NEWS-20100914052420163-00159-00177-ia360914/TOMBSTONE
    ... (repeated many times) ...

[siznax (siznax)]

this also effects the speed of processing as the number of "finished" uploads increases.

each pass over the prerequisites (PACKED, MANIFEST, LAUNCH/TASK, TOMBSTONE) can take a while on a busy crawler, and can make for unnecessarily large log files if you tee draintasker stdout/err to a file.

i've created a script to move aside tasks containing a TOMBSTONE file to an "_done" directory alongside the xfer_job_dir, e.g.

xfer_dir => /1/incoming/newscrawl
done => /1/incoming/newscrawl_done

this can speed processing, and make logs smaller.

we may want to include this in normal processing. if so, we'll want to also update "get-status" to look for this _done series.

[siznax (siznax)]

oops, the script is:

eject_tombstones.sh
------------------------------------------------------------------------
r48 | steve | 2010-11-18 00:05:56 +0000 (Thu, 18 Nov 2010) | 1 line

move TOMBSTONES out of the way of drain processing