wanted: recompile system when dependency changes

[Tobias C. Rittweiler]

As far as I'm told (and that matches my experience), ASDF does not recompile
the current system if a dependent system changed, unless of course :FORCE T
is specified.

[Robert P. Goldman (rpgoldman)]

Don't you mean "a depended-on system" rather than "a dependent system"?

[Nikodemus Siivola (nikodemus)]

At the very least we should document the behaviour IN LARGE TYPE.

I don't think we should magically recompile things which depend on FOO when we it changes, but when the up-to-dateness of those things is next checked via eg. LOAD-OP we should, IMO.

Changing this should not break anything as far as I can see.

[Robert P. Goldman (rpgoldman)]

This is harder than one might like to fix, because of oddities in TRAVERSE and in the object system of ASDF (notably the fact that the SYSTEM object represents, at one and the same time, BOTH the system itself, with components, AND any additional operations that must be done AFTER the system's components are loaded).

For that reason, I propose to push this to ASDF 2.1.

[Tobias C. Rittweiler (tcr)]

I hope that an 2.1 will ultimatively come out even after Fare resigning
after 2.0. I suggest to make this a priority item for 2.1.

I'm working on a system which is split up in many systems, and it
totally sucks that I basically always have to FORCE loading of the
system which also recompiles a whole bunch of third party code.
Using SBCL, the compilation times are starting to annoy me.

[Faré (fahree)]

My resignation is taking some time to happen, it seems -- I'm waiting for 2.0 to be adopted by all implementations, which may take a few more months, and a few bug fixes. Now at 2.006.

To come back to the topic of this bug,
* not just the current system, but any of its dependencies are also affected by this lack of recompilation on modified dependency.
* I think it's a bug that should be fixed.
* I don't understand at this point what happens in subsequent runs where some systems have been updated after their dependents have been loaded, then you refresh a dependent.
* I think rpg is being overly conservative in being afraid to fix this bug.
* The "use case" I know for justifying this bug is that people who like being able to update, say, CLX, to fix some bug, without having to recompile EVERY subsequent system in their CLIM application. I say this calls instead for finer grained dependencies, and a system capable of correctly handling incremental compilation - i.e. XCVB.

[Peter Keller (psilord)]

I'm developing a library intended to be used by other applications and I include some
example applications. The library is a system of its own (suppose A) and
one of my example applications is another system (suppose B) which depends upon A.
I changed a macro definition in A, then (require B) and I saw A recompile (and
stopped paying attention), and then wondered why my application wasn't working like
how I expected. It took several hours to figure out what happened with odd periods
of sometimes it worked, and sometimes it didn't.

I came to understand that system B would never be recompiled even though A, upon
which it depended, changed. I also came to understand that this is somehow a desired
feature. The reason why it worked is when I adjusted B with debugging output and it
finally got recompiled. Then when I removed the debugging and kept modifying A, thinking
I understood the problem, it stopped working. This was hair raising in my frustration.

I truly believe the current behavior to be a wrong default and that if a system depends on
another one, and the one changes, then both should be recompiled as expected.

Any other behavior is tantamount to changing a C++ object definition, recompiling
the library which holds the object's cpp files, and then simply relinking the executable
which used it instead of recompiling the executable which used the header file and now
the size and locations of things in the object are skewed between the library and the application.
This is a recipe for disaster and undefined behavior.

I'd much rather wait 10 minutes to recompile a pile of code than waste several hours
debugging phantom problems.

[Faré (fahree)]

Anton Vodonosov was asking on the mailing-list about what turns out to be this bug.

Now that :force AND :force-not are both working, users have a means to override the default behavior, and it matters less what the default behavior is. Therefore, I believe it is better that we now fix this bug and do the Right Thing(tm), which is recompiling the systems that depend-on one that has changed.

This is worthy of a release of its own, so I propose we wait until I release 2.23 next week, then implement, test and release a fix as 2.24 right afterwards.

[Faré (fahree)]

OK, I think it is time to address this old bug indeed.

[Faré (fahree)]

Fixed in ASDF 2.26.8, together with another cleanup of TRAVERSE and OPERATE,
also fixing some FORCE / FORCE-NOT bugs along the way.

People who want the old behavior can use FORCE-NOT to emulate it cleanly.

[Robert P. Goldman (rpgoldman)]

Great. I would like to suggest two sub-tasks for fixing this bug:

1. In order to decide whether a downstream system has to be recompiled, we need to know what was the state of the upstream system when the downstream system was compiled and loaded. In general, ASDF senses this sort of thing by examining the state of the filesystem. Since ASDF does not generate a single artifact, the way a makefile does, we may need to add some sort of entity to the filesystem to store the required information.

2. It will be a lot easier to manage this task if we have a better model of what it means to operate on a system than the current "after method" semantics. For backwards compatibility, we may need to add a new op to the object model, since COMPILE-OP and LOAD-OP on MODULE are already taken for "after method" type purposes.

[Faré (fahree)]

I suppose you started writing before you saw my commit message.

Is there anything wrong with my fix?

I believe you're trying to fix other issues:
1- maintaining coherence when some system changes and that other loaded systems depended on it that are NOT dependencies of the system being currently loaded. As long as they are, there is no problem.

2- What if A, system or not depends on B, and A was recompiled in a previous session, but not B, causing B to miss a modified macro definition in A? The problem is easier to reproduce between systems (compile B, modify A, compile A, compile B), but a compilation of a single system interrupted by an error can make it happen even within a single system. It's a bug whereby from dependency to dependency we propagate a flag that means "needs to be recompiled in current image", rather than a timestamp (or content digest) saying "needs to be recompiled if not up-to-date with respect to THIS". Will you report that bug? Create a test case?

That's worthy of another bug, I believe. Within a system and/or if we don't omit system dependencies we could wipe all obsolete output-files at the beginning of the perform-plan. Meh.

3- Time to invest more efforts into XCVB, again.

[Robert P. Goldman (rpgoldman)]

Yes, you are right: my response was composed after your message about this being the time to fix the problem and before I read your follow-on that you had committed a fix.

I looked over your patch, but didn't have time to think about it deeply. Unfortunately, I won't have time to really dig into it until sometime late next week...

I think my concern is about your issue # 2: what if the dependency change is not contained within a single running lisp image. E.g., what if the upstream system was recompiled in process 1, which is then terminated. Now I start process 2, and load a downstream system. How do we know that the downstream system was compiled before the upstream one was modified?

I have a hazy notion that I looked into this, and that the existing ASDF logic was not obviously extensible to solving this problem. I was toying with the idea that we might want to write files into the filesystem indicating the compilation date of the systems. That would give us a persistent marking of the state of the dependency graph. But I didn't think deeply enough about this to prove to my satisfaction that this was sufficient.

So is this your #2? I could try to write up a ticket about that, if so.

[Faré (fahree)]

Let's discuss this issue #2 in its own proper bug:
https://bugs.launchpad.net/asdf/+bug/1087609

[Faré (fahree)]

Released in 2.27.