failure to check timestamps of dependencies

[Faré]

What if A, system or not depends on B, and A was recompiled in a previous session, but not B, causing B to miss a modified macro definition in A? ASDF fails to recompile B.

The problem is easier to reproduce between systems (compile B, modify A, compile A, compile B), but a compilation of a single system interrupted by an error can make it happen even within a single system.

It's a bug whereby from dependency to dependency we propagate a flag that means "needs to be recompiled in current image", rather than a timestamp (or content digest) saying "needs to be recompiled if not up-to-date with respect to THIS". Will you report that bug? Create a test case?

Within a system and/or if we don't omit system dependencies we could wipe all obsolete output-files at the beginning of the perform-plan. Meh. That's ugly.

We could instrument ASDF to pass stamps around, but it might not be compatible: operation-done-p and mark-operation-done would have to be replaced by something very different. Is the incompatibility worth it? How many extensions are there that use these?

Originally from https://bugs.launchpad.net/asdf/+bug/479522/comments/11

[Faré (fahree)]

I believe this is as important a bug as it can get: a failing in the fundamental way that ASDF computes and propagates dependency changes.

[Faré (fahree)]

My current feeling is that a special variable *stamp* should hold the "latest known timestamp" involved in current action's dependencies. Also, we should base such stamp purely on filesystem write-date's, not at all on get-universal-time.

I've started work on that. My solution-in-progress requires backwards-incompatibilities, but I believe these can be reverted by my currently making *stamp* a special variable.

[Robert P. Goldman (rpgoldman)]

I agree about making everything depend on filesystem write dates.

What do you think about my idea about dropping a dummy file into the filesystem to record the last compile date of systems? Is that unnecessary?

I suppose one can simply traverse the current system's files to do the checking, and compute a MIN over the compiled file dates as the system freshness date, compared to the MAX of the write dates of the parent system.

My dummy file was an attempt to avoid this extra work. You have obviously been thinking harder about this, so I defer to you.

Question: is there any case where there's an aspect of system state that cannot be done by file write date? E.g., reasoning about the in-memory state of a file that is simply loaded from source instead of compiled (we have a system like this, involving processing of a DSL into in-memory data structures, and it's a nightmare).

[Faré (fahree)]

Yes, I was thinking about simply accumulating the max timestamp of all files input or output before a given file is reached, and the timestamps of direct subcomponents as well.

Then, writing the code, I see that it makes proper do-first treatment much harder (having to revert to kludges remindful of ASDF 1); on the other hand, it might also make do-first wholly unnecessary, since performing operations will give them a timestamp based on the input and output files, as opposed to the load-time stamp. And so I'm thinking of plainly getting rid of do-first. Do you think there's any problem with that?

[Faré (fahree)]

OK, after way too much work, including a major simplification of traverse, introducing and painfully reverting some backwards-incompatibilities, and much debugging, I have something that looks like it works.

I even had to fix one test you (rpgoldman) wrote long ago, in which the new ASDF *improves* the behavior in a situation you contrived using internals: test-module-excessive-depend.script

Please test ASDF 2.26.9 or later. I may do a pass of cleanups later.

[Faré (fahree)]

Released with ASDF 2.27.