Suggestions for improving version management

[Didier Verna]

This is a proposal for improving the way ASDF handles systems/components versions. The problem is that it currently forces systems to follow its own (simplistic) versioning scheme. The idea is to remove that constraint, letting system authors version their software as they wish, without breaking backward compatibility. In case a system uses a non default versioning scheme, then ASDF should delegate the versioning logic to the system itself.

ASDF currently does 2 things with version information: checking for version compatibility and normalizing version strings (in its own, arbitrary way).

Checking for version compatilibity is done via VERSION-SATISFIES. The good thing is that this already is a generic function, so delegation is possible by subclassing the ASDF:SYSTEM class and providing a new method specialized on this new class (see below).

Normalizing version strings is done via NORMALIZE-VERSION. Unfortunately, this is a regular function. The attached patch implements a solution. It provides a new generic function NORMALIZE-COMPONENT-VERSION, with a default method that wraps around NORMALIZE-VERSION. Systems authors could then provide additional methods for their own system subclasses, implementing their own normalization scheme or simply leaving the version slot's value as-is.

With this in place, a system author could implement his own versioning scheme like this:

(defclass my-system (asdf:system) ())

(defmethod asdf:version-satisfies ((system my-system) version)
  #| implement your logic here |#)

(defmethod asdf/parse-defsystem:normalize-component-version
    ((system my-system) form &key pathname component parent)
  (declare (ignore pathname component parent))
  #| normalize as you wish, or just return FORM|#)

(asdf:defsystem :my.system
  :class my-system
  :version '(however you wish to store it)
  #|...|#)

On top of that, Pascal Bourguignon suggested that we maintain both a raw (canonical) version slot and a human-readable one. Names could be :version / :version-string, :short-version / :long-version or whatever else...

[Robert P. Goldman (rpgoldman)]

How about we provide a FORMATTED-VERSION (probably a better name is needed) generic function that the programmer can specialize in order to format otherwise hard-to-read contents of the VERSION slot?

[Didier Verna (s0-adm-ph)]

Maybe AESTHETIC-VERSION ?

[Robert P. Goldman (rpgoldman)]

Much better!

So we'd have the following protocol:

(defgeneric version-satisfies (component version))
(defgeneric normalize-component-version (system form &key pathname component parent))
(defgeneric formatted-version (component version))

Question: should we also have

(defgeneric version-geq (component version1 version2)
  (:documentation "True if VERSION1 is greater than or equal to VERSION2."))

I'm a little worried about VERSION-SATISFIES, because I would like to see us support both
(:VERSION <v>)
and
(:VERSION <min> <max>)
[I just made a backward incompatible version of XMLS]

and I don't think that programmers should necessarily have to write their own logic for V >= K and K1 <= V <= K2.

In this case, programmers would generally leave VERSION-SATISFIES alone and it would be sufficient to write your own VERSION-GEQ.

[Faré (fahree)]

What about, simply, VERSION-STRING instead of FORMATTED-VERSION ?

As for comparison, what about

(COMPONENT-VERSION-COMPATIBLE-P SYSTEM REQUIRED-VERSION PROVIDED-VERSION)

?

[Robert P. Goldman (rpgoldman)]

I'm ok with compatible-p so long as we also have a generic VERSION-GEQ, so that programmers who want to simply have different format version specifiers, but still want to support (:VERSION X) and (:VERSION MIN MAX) don't have to write their own logic (per last comment). I think, for example, Pascal's case might be handled just by implementing VERSION-GEQ, and not bothering with COMPONENT-VERSION-COMPATIBLE-P

VERSION-STRING seems possibly misleading if the version generally is a string. I like Didier's AESTHETIC-VERSION (or perhaps AESTHETIC-VERSION-STRING?) because it makes the purpose of the function clear in the name...

A follow-on question: is there any reason to have this as COMPONENT-VERSION-COMPATIBLE-P instead of SYSTEM-VERSION-COMPATIBLE-P?

It seems like we imply that individual components can be versioned separately, but I have no reason to believe that this is actually implemented or, if it is implemented, implemented correctly. The use case seems obscure, and fussing around to get it working seems like a waste of energy.

[Faré (fahree)]

What about VERSION-DISPLAY-STRING ?

And maybe you're right that compatible-p is a feature that no one intends to use anyway — the right thing if you're going to be incompatible is to change the system name, e.g. name your system FOO2 instead of FOO, which also allows the two things to coexist. Then, (SYSTEM-VERSION<= SYSTEM VERSION1 VERSION2) ?

Also, instead of (:VERSION MIN MAX), shouldn't it be (:VERSION START END) when the condition is

(AND (SYSTEM-VERSION<= START VERSION) (NOT SYSTEM-VERSION<= END VERSION))

[Faré (fahree)]

I meant

(AND (SYSTEM-VERSION<= SYSTEM START VERSION) (NOT SYSTEM-VERSION<= SYSTEM END VERSION))

[Didier Verna (s0-adm-ph)]

One thing that is still obscure to me is whether this whole version logic things applies to random components, or only systems. Do people actually version sub-components ? For instance, the normalization part is part of the parse-defsystem package. Does that apply to all component types ?

As for human readable versions, I still prefer my aesthetic-version over the other proposals, but that's not really important.

[Robert P. Goldman (rpgoldman)]

Re #9, this is what I was referring to in #6: some of the terminology seems to suggest that arbitrary components can be versioned, but I believe that there is, at most, only a fragment of the relevant logic available.

I'm not excited about the code proliferation that would be required to extend this to arbitrary components, especially absent any pull from the user community or even a plausible rationale.

I'm ok with either AESTHETIC-VERSION or VERSION-DISPLAY-STRING.

And Fare (#7) is quite right: the MIN-MAX is a half-open interval [MIN,MAX), where the version must be >= MIN and < MAX.

[Pascal J. Bourguignon (pjb-informatimago)]

> On 16 Jun 2015, at 22:51, Robert P. Goldman <email address hidden> wrote:
>
> Re #9, this is what I was referring to in #6: some of the terminology
> seems to suggest that arbitrary components can be versioned, but I
> believe that there is, at most, only a fragment of the relevant logic
> available.
>
> I'm not excited about the code proliferation that would be required to
> extend this to arbitrary components, especially absent any pull from the
> user community or even a plausible rationale.

It seems to me that the unit of distribution is a set of related asdf systems bundled together (in a tarball or git repository).

It is meaningful to give version numbers to asdf systems, so we may detect new "releases", and possibly eventually, so we may indicate that a system depends on another system in a specific version or range of versions.

However, I fail to see the point of giving version numbers to components: there's an implicit dependency between the system and the components it refers, within its distribution unit. Configuring a dependency between a system and a specific version of a component could require downloading and installing two versions of the same system with cross dependencies. This is most certainly a bad idea.

If one has a complex system with components that can evolve separately, then those components should be distributed as separate systems, and the dependency should be declared between systems. Then one system can declare a dependency with a specific version or range of a "sub-" system.

So indeed, versions should be attached only to systems, and a distribution unit may contain related systems (with possibly different versions. eg. you can have :mylib version 1.2.1, :mylib-test version 1.2.0, and :mylib-example version 1.0.3 in the same tarball. This should provide fine enough a grain and sufficient confusion for anybody, without trying to give version numbers to non-system components.

--
__Pascal J. Bourguignon__

[Faré (fahree)]

ASDF 1 historically allowed for version numbers in components. While almost no one uses this feature, I remember seeing with horror that some systems in Quicklisp were annotating components with version numbers, while fixing another issue with ASDF -- a run of cl-test-grid with a modified ASDF would tell for sure if that's still the case. Sigh.

I don't condone the use of :version for anything but a system. I believe it's a bug that ASDF allows it, and that ASDF can't conceivably do anything useful with it, nor is it ASDF's job to handle file version within a repository. That's what git is for.

Still, removing this feature is a backward-compatibility hassle, and requires hard work from whoever is going to make that change possible: run cl-test-grid, get all offenders to fix their .asd files, and if it happens in an unmaintained system, fork it and fix the fork.

[Faré (fahree)]

Moving :version to system rather than component should probably be a separate bug.
COMPONENT-VERSION should then be deprecated in favor of SYSTEM-VERSION or some such.

Le sigh.

[Robert P. Goldman (rpgoldman)]

Agreed that moving version should be a separate bug.

[Didier Verna (s0-adm-ph)]

Something that just occurred to me: the good thing with this proposal is that it is fully backward compatible.

1. Systems already honoring ASDF-style version strings are not affected.

2. For other systems like mine, ASDF already complains, and uses nil. One would end up creating methods for generic functions that may not exist until the proposal is implemented, which is not an error as Lisp automatically creates them. So you would get a bunch of additional warnings but that's essentially it.