Archive Widecrawl

  1. Bug #667482
  2. Comment #1

Comment 1 for bug 667482

Revision history for this message
siznax (siznax) wrote : Re: need less verbose logging option

this also effects the speed of processing as the number of "finished" uploads increases.

each pass over the prerequisites (PACKED, MANIFEST, LAUNCH/TASK, TOMBSTONE) can take a while on a busy crawler, and can make for unnecessarily large log files if you tee draintasker stdout/err to a file.

i've created a script to move aside tasks containing a TOMBSTONE file to an "_done" directory alongside the xfer_job_dir, e.g.

xfer_dir => /1/incoming/newscrawl
done => /1/incoming/newscrawl_done

this can speed processing, and make logs smaller.

we may want to include this in normal processing. if so, we'll want to also update "get-status" to look for this _done series.