option to clear missing or invalid images

[danh]

Jude told me about what has been a known problem since at least Eric's time:

Under some circumstances (such as when a scribe gets powered down during image processing) there will be an error file. The file will have a name like ERROR_JP2_PROC_EXIF_0021.

When this happens, the scriblio software will refuse to let anything happen with the book.

Hank, Jude, and i have discussed the problem, and one possible solution would be for the scribe software to offer the option of repairing the the damaged book by finding each defective image and replacing it with a stock image which says "RESHOOT ME". We would also mark the image in the republisher as needing to be reshot, so that it would not be checked in if it doesn't actually get reshot.

[danh (danh-archive)]

Eric actually documented this issue, and the link (from Callie) is:
http://home.us.archive.org/cgi-bin/twiki/viewauth/BooksGroup/EricOstlundNotes

[Hank Bromley (hank-archive)]

The plan sounds fine to me; I just wanted to add a little more background.

Steve added the code to create these temp files (and delete them when an image was processed successfully) in the early days of Scribe2, when we were getting a lot of broken books uploaded, with missing or 0-length jp2s. The temp files were intended to flag a broken book at an earlier stage than the derive, which is when they were being found.

The actual cause of the botched processing was never found; once Steve's changes (these temp files, plus another set of changes that noticed if jp2s were still being created and forced the operator to wait until the processing finished) reduced the number of broken books being uploaded, the sense of urgency was reduced.

I compiled what I thought was compelling evidence that the forked processes for creating jp2s were somehow being duplicated, creating each file twice, and causing errors when the second process caught up to, and stepped on the toes of, the first process (e.g., the second might starting writing to a given file before the first was done with it), but never had the opportunity to follow up on it.

During the 2-1/2 years since then, Scribe2 has been working "well enough" to leave it alone. But if, for whatever reason, the error rate with Scribe2 is rising again, we might want to re-open that investigation. I had some pretty simple ideas for how we could record the total number of file-creations happening across all processes. That would tell us pretty quickly whether my conjecture was correct - the number would be approximately 2N instead of N.