system goes idle, but there is data to process

[John Lekashman]

Something in the timers for retry isn't quite right.
So, it will probably finish in some period of time.

However, there should be a concept of: This system is to be
emptied, don't wait up. We have this issue frequently enough
that it should occur. In prior incarnations of the system, I could
just turn the sleep between checks down, and the system would execute faster.

At this point in time, I have on the queue, absolutely no tasks waiting on anything.
Nothing waiting, nothing running.
Yet there is 200G of data, sitting there to be drained, and things are idle.

for i in 35 36 38 40 42; do ssh ia3609$i du -sh /{1,3}/incoming/wide; done
29G /1/incoming/wide
68G /3/incoming/wide
61K /1/incoming/wide
57K /3/incoming/wide
19G /1/incoming/wide
9.5G /3/incoming/wide
47G /1/incoming/wide
31G /3/incoming/wide
19G /1/incoming/wide
426K /3/incoming/wide

Now, I can wander around from directory to directory, and edit the retry timer to say, do it now.
But, something to do that more cleverly would be appropriate.

[siznax (siznax)]

does this issue only apply to items which have been stalled by RETRY logic?

[siznax (siznax)]

if so, i think implementing an "override-retry" config param wouldn't be too difficult. maybe if that param were set to 1, then any extant RETRY files could be ignored and processing might proceed as normal.