update filter order

[paul.n]

Request to change filter order.
New and Current orders are listed below.

Thanks!

New order:

'upload not allowed'
'subdirectory'
'illegal identifier'
'unexpected location'
'item (bibliographics) not found on cluster'
'no frames'
'no file extension'
'images already uploaded for this item'
'no images'
'busy' (or, if a system problem: 'could not check status')
'images are jpgs' (unless allowed)
'both jp2 and jpg'
------------ > at this point, if it is split, we let it through
'no qpf'
'no idf'
'no previews'
qpf ' exists but is not a dir' (system type problem)
'could not make dir ' qpf (system)
'unexpectedly no qpf' (system)
'malformed qpf name ' qpf (system)
'unexpected parse error on qpf name ' (system)
'rsync qpf failure'
'Could not parse ' qpf
'frame count ' fcnt ' vs. qpf count ' qcnt

Current Order:
> 'upload not allowed'
> 'subdirectory'
> 'illegal identifier'
> 'unexpected location'
> 'no frames'
> 'no images'
> 'no file extension'
> 'item (bibliographics) not found on cluster'
> 'images already uploaded for this item'
> 'busy' (or, if a system problem: 'could not check status')
> 'images are jpgs' (unless allowed)
> 'both jp2 and jpg'
> ------------ > at this point, if it is split, we let it through
> 'no qpf'
> 'no idf'
> 'no previews'
> qpf ' exists but is not a dir' (system type problem)
> 'could not make dir ' qpf (system)
> 'unexpectedly no qpf' (system)
> 'malformed qpf name ' qpf (system)
> 'unexpected parse error on qpf name ' (system)
> 'rsync qpf failure'
> 'Could not parse ' qpf
> 'frame count ' fcnt ' vs. qpf count ' qcnt

[danh (danh-archive)]

Per email from Paul, the reason for the change in order is that some
decisions to process items on the machines are made on the basis
of whether the item has been loaded (i.e., has bibliographics
in the cluster).

The current order is based on the simple optimization that we won't
check the cluster if there are no frames on the microfilm machine.

In the new order, we will know earlier in the cycle when something
is not loaded, which will help with the decision making.

[paul.n (paul-n)]

The staff would really like "no images" to go below:

'no qpf'
'no idf'
'no previews'

But I am unsure how this will interact w/ the split_image requirements

Would it be possible to put 'no images' below those notes but also not allow split_item reels to go through (if they don't have images)???

[danh (danh-archive)]

We've had a fix out for this for quite a while, so marking the bug closed.

If we want a different filter order, then we'll open a new bug report.

[paul.n (paul-n)]

Sorry to open this one up again!

We have a sneaking suspicion that some reels are getting the wrong diagnosis. This is the scenario we are concerned about:

1. A reel gets uploaded
2. The processing person comes back to that computer and looks at the uploader
3. At that precise moment the contents of the reel uploaded in step one are being deleted and presents a "no frames" error
4. The processing person deletes this item, causes a redrow, and files a bug, requesting that the item be rescanned
5. Hank comes along and see this redrow. Since the necessary files have been uploaded, he asks the processing person to delete the directory (it's already gone) so he can push the reel along
6. The second upload fails due to "Item already online" message

I think this could be avoided by moving "busy" up on the filter order.
Is there any problem with moving "busy" above "no frames" and under "item (bibliographics) not found on cluster"?

[Hank Bromley (hank-archive)]

I'm not sure that scenario is possible. (We can talk more about why, if we need to investigate more fully.) But let's assume it is - then if it did happen, the item should have two bugs filed for it, one (from the "no frames" error) with a request for the rescan, and one (from the archive.php redrow) with a request to delete the original files.

So do we have items for which two such bugs have been filed?

[paul.n (paul-n)]

  Yes, I suppose actual proof of the problem would be good!
I took a brief look at the bugs and could not find any pair of bugs that
fit the scenario outlined in this bug.
I spoke with Sam and Jesse a bit more and it seems like we are stumped
on why we end up with so many "item already online" notices in the
uploader (meaning we unintentionally scanned an item twice). We've been
getting about one of these a day, sometimes more.

The position of "busy" in the upload filter led to the suspicion that
the scenario was possible but I'm not sure if it actually happened.
The other hypothesis, maybe more probable but also requiring
investigation, is that the physical reel is simply put on the wrong
shelf ("to-be-scanned instead" of "transferred")

On 8/19/10 6:32 PM, Hank Bromley wrote:
> I'm not sure that scenario is possible. (We can talk more about why, if
> we need to investigate more fully.) But let's assume it is - then if it
> did happen, the item should have two bugs filed for it, one (from the
> "no frames" error) with a request for the rescan, and one (from the
> archive.php redrow) with a request to delete the original files.
>
> So do we have items for which two such bugs have been filed?
>

[Hank Bromley (hank-archive)]

Next time you get an "item already online" notice, let me know, and I'll see what I can figure out about the history of that item.