Been meaning to reply to Paul's initial report since it was posted. Sorry for being so slow!

Our use of repub_state is indeed all confused, due to piecemeal changes over time, and is ripe for being totally rethought. At one point, repub_state of 6 was used to indicate the book had finished deriving, and some microfilm pathways still produce that result - we have 125K items with repub_state=6. (Actually, having just checked the code, the 6 is being set on microfilm at the end of the postprocessor book-op, and staying that way after the derive for reasons that will become clear in the next paragraph.)

Then at some point a function was added to the deriver code and set to run at the end of each derive, called nonMicrofilmBookSetRepubState4(). It changed repub_state to 4 for anything that had mediatype=texts, and wasn't microfilm. The comment introducing the function says:

  // If you are a book but not a microfilm book, then set your 
  // repub_state to 4.  This is meant to be invoked near the 
  // end of a derive, so that it is not done if the derive 
  // has bombed out earlier.  It is for the benefit of scribe 1 books. 

For scribe 2 books, the modify_xml.php Paul initially asked about gets queued as a "next_cmd" for the archive.php. That meant we were setting repub_state to 4 twice for scribe 2 books, once at end of the derive, and again in the follow-on modify_xml. Once we began redrowing modify_xml's that produced no change, all the follow-on modify_xml's began redrowing. At that point (April 2010) I tweaked nonMicrofilmBookSetRepubState4() to skip the repub_state change for scribe 2 books, as a short-term solution, pending clarification of the whole mess. My comment in the code reads:

    // FIXXX - April 2010, we need to clean up repub_state values and how 
    // they're set - no one seems clear on why we set repub_state to 4 after 
    // deriving (shouldn't it be 6?), nor why we do it both here and via the 
    // separate modify_xml.php task for Scribe 2 books. The modify_xml is now 
    // redrowing due to repub_state already being set to 4 here; as a short-term 
    // solution, skip it here for Scribe 2 

    // (probably want actually to drop the extra modify_xml task and set 
    // repub_state here for all texts items [to 4 or to 6? still unclear])

And, of course, I never got around to starting the discussion I intended to, in order to get that all clarified, so that's where things remained for a while.

Then, in an effort to reduce the number of green rows, Tracey noticed that we had all these modify_xml tasks hanging around, green, for the duration of the derives on their items. Well, that seemed pretty silly, given how fast modify_xml's are, compared to derives. So she added an automatic priority adjustment that caused the modify_xml's to run before the derives, rather than after, the way it used to (which had been causing the redrows that my previous code change had been a response to).

So what we have now seriously makes no sense at all.

That means we get to decide what we should do instead!

I have no strong feelings on what repub_state we should use to signify that an item has finished deriving, beyond wanting us to be consistent about whatever it is. But I do think, once we decide what that value is, we should probably dump the extra modify_xml task, and have nonMicrofilmBookSetRepubState4() set that value for all books at the end of the derive, whether scribe 1 or scribe 2 or microfilm.

Now that OBF is no longer in use, there may also be no reason for the preprocessor and postprocessor book-ops to be changing repub_state, but I'm less sure about that.

In any case, let's figure out what we actually want the repub_states to be at each stage, and then we can fix the code the make it happen, with a minimum of redundancy and confusion.