CCL implementation-identifier - include revision and source control revision

Bug #1366379 reported by Anton Vodonosov on 2014-09-06
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ASDF
Undecided
Unassigned

Bug Description

For CCL 1.10 release candidate (not a release) we have:
  (asdf:implementation-identifier) => "ccl-1.10-f96-linux-x86"
  (lisp-implementation-version) => "Version 1.10-RC1-r16188M (LinuxX8632)"

For CCL 1.9

(asdf:implementation-identifier) => "ccl-1.9-f96-linux-x86"
(lisp-implementation-version) => "Version 1.9-r15756 (LinuxX8632)"

I see two problems here:

1. ASDF does not distinguish between release candidate and proper release.
    Note how CCL's (lisp-implementation-identifier) adds RC1 to it's result.

2. CCL releases are not concrete versions. As far as I understand,
   depending when you install CCL release, you may get different versions,
   because developers publish bug fixes. So ccl-1.9 does not specify
  CCL version precisely.

I suggest to patch asdf:implementation-identifier to include both
what CCL maintainers call revision - RC1, and source sontrol revision number - r15756.

The attached patch is based on CCL's implementation of cl:lisp-implementation-version

Anton Vodonosov (avodonosov) wrote :
summary: CCL implementation-identifier - include revision and source control
- revition
+ revision
Faré (fahree) wrote :

The purpose of implementation-identifier is to distinguish between incompatible ABIs, hence the f96 there, identifying the fasl version number. Should it or should it not include more information about the implementation? That's not obvious.

If you include svn revision, you don't need to include RC1, that's redundant. There's a good reason to not include redundant information: pathname size limitations on some implementations / operating systems / filesystems. The size limitation was notably hit on some Lisp implementations running on Windows.

Note that the source control revision number *ought* to be enough to identify precisely the source code used, but isn't, because ccl uses unpinned externals, and so unless you take extra precautions, it will pull in parts of the code at HEAD instead of the given version. Maybe I should release the CL script I once wrote at ITA to properly compile CCL...

Robert P. Goldman (rpgoldman) wrote :

To follow-up on Fare's point, Anton -- I know that CCL versions may have different patch levels, but surely they don't have incompatible FASLs, do they?

Allegro, by comparison, had 8.2 for a very long time, but the fasls remained consistent, although one's lisp could vary pretty extensively in the set of patches that had been applied...

Anton Vodonosov (avodonosov) wrote :
Download full text (3.2 KiB)

Robert, Fare

I've considered the aspects you describe before bothering
your with this proposal.

1. As for strengthening CCL version precision:

   Yes, fasl file format is reflected as -f86 in
   asdf:implementation-identifier for CCL. And the fasl
   format is not necessary changed between CCL releases,
   so the fasl format remains parsable/loadable by both
   ccl-1.9-f96 and ccl-1.10-f96.

   But the code stored in that file format may still be broken
   due to compiler bugs, so we may want to recompile on
   new compiler.

   Especially, if the old compiler was NOT a release, but an intermediate
   development version or a release candidate - I would want to recompile
   when I switch from 1.10 release candidate to proper 1.10.

   ASDF already segregates fasls of the same -f86 format, built
   by different releases. Example cache directories:

   ~/.cache/common-lisp/ccl-1.10-f96-linux-x86/
   ~/.cache/common-lisp/ccl-1.9-f96-linux-x86/
   ~/.cache/common-lisp/abcl-1.3.0-fasl42-linux-x86/
   ~/.cache/common-lisp/abcl-1.3.0-svn-14683-fasl42-linux-x86/
   ~/.cache/common-lisp/ecl-13.5.1-unknown-linux-i686/ ;; release 13.5.1
   ~/.cache/common-lisp/ecl-13.5.1-237af2e8-linux-i686/ ;; git version after release 13.5.1
   ~/.cache/common-lisp/sbcl-1.2.0-linux-x86/
   ~/.cache/common-lisp/sbcl-1.2.0.69-615be91-linux-x86/

   Unlike the other lisps in the above example,
   CCL's asdf:implementation-identifier never has source control revision,
   even in trunk versions. Moreover, for the other lisps the release
   version means exact source code version, while
   in CCL version 1.10 allows wide variety of
   versions, including release candidates.

   I admit that Allegro also updates itself
   by patches, and this information is omitted
   in implementation-identifier:

   ~/.cache/common-lisp/acl-8.2-linux-x86/
   ~/.cache/common-lisp/acl-9.0-linux-x86/

   But at least Allegro release candidate will not be
   named same was as a proper release (I hope).

   Therefore, I think it make sense to strengthen
   CCL implementation identifier, and svn revision
   is the only precise enough attribute for all cases.

   But I don't insist, it's just a proposal for your consideration.

2. Is -rc1 redundant if we use svn revision number.

   Only matters if you consider (1) convincing, and agree
   to strengthen CCL's implementation identifier.

2.1 Consider -rc1 as a part of human readable version number:
    1.9, 1.10-rc1, 1.10.

    If SVN revision number makes -rc1 redundant, then
    the 1.9 and 1.10 numbers are technically redundant too.

    But the human readable version number is convenient - it is
    good to understand immediately what of my lisps created that fasl directory.

2.2 The long path problem on Windows indeed exists, I"ve been bitten
     by it. But the fasl path is long because it consists of the path to the
     cache directory + the full path of the original source:

    /home/anton/.cache/common-lisp/ccl-1.9-f96-linux-x64/home/anton/prj/cl-test-grid/cl-test-grid2/agent/agent.lx64fsl

    -rc is just 3 letters, not a significant addition.

2.3 The -rc only appears in release candidate, normal
     releases don't have ...

Read more...

Anton Vodonosov (avodonosov) wrote :

Fare wrote:

> Note that the source control revision number *ought*
> to be enough to identify precisely the source code used,
> but isn't, because ccl uses unpinned externals, and so
> unless you take extra precautions, it will pull in parts of the
> code at HEAD instead of the given version. Maybe I should
> release the CL script I once wrote at ITA to properly compile CCL...

I think it makes sense to record this info as a CCL ticket.

Faré (fahree) wrote :

svn revision is not enough: you need to know which BRANCH you're on, and that's why the major / minor version number is important even with svn revision information. When this information is available, indeed the -f96 fasl version information becomes redundant, as well as any -rcX information.

If you open a ticket, I'll attach my release script to it. Unhappily, I seem to have lost my password to my clozure trac account, and the automated retrieval system fails to send me another password. Oops. I'll ask rme to reset my password.

Robert P. Goldman (rpgoldman) wrote :

One question and comment:

Q: are you proposing to add only the release candidate number to the ASDF implementation specifier?

Comment: I'm reluctant to complicate the CCL version further. I think the key issue is your point number one:

   But the code stored in that file format may still be broken
   due to compiler bugs, so we may want to recompile on
   new compiler.

   Especially, if the old compiler was NOT a release, but an intermediate
   development version or a release candidate - I would want to recompile
   when I switch from 1.10 release candidate to proper 1.10.

I'm inclined to say "if you use a release candidate, rather than a release version, you are accepting some dangers, and ASDF need not necessarily protect you from them.

It's like using make with a prototype version of GCC: if you are a gcc developer, you should know that you are on risky ground and may need to manually smash your object files.

So the question would be -- is there some reason that using CCL release candidates, by people who are likely to be tripped by fasl/compiler incompatibilities is a common enough use case that we should complicate the ASDF core to deal with it?

I know that you are doing extensive testing, which is a big service to the community, and one we should encourage. So a follow-up question:

Should we make the implementation-version hookable so that you could make a contrib for the purposes of cl-test-grid that would give you exactly the information you need and want?

Anton Vodonosov (avodonosov) wrote :

A: That's possible too.

     I originally proposed to include both the release candidate number and svn revision, because I THINK official CCL releases may be updated by patches without changing release number.

As for the complication of ASDF. Currently ASDF uses two CCL variables when formatting implementation-identifier. The change is to use 2 more variables.

> Should we make the implementation-version hookable so
> that you could make a contrib for the purposes of cl-test-grid
> that would give you exactly the information you need and want?

No need, thank you.
cl-test-grid uses code copy/pasted from ASDF for lisp implementation identifier, I've already fixed updated code in cl-test-grid.

Faré (fahree) wrote :

CCL records all changes in svn, by cutting a new branch for each release, see e.g. the latest
http://trac.clozure.com/ccl/log/release/1.10
or the previous
http://trac.clozure.com/ccl/log/release/1.9

So the branch number (major.minor) plus svn revision includes any -RC1 or -f96 information and more. Note that because of the way CCL bootstrapping works, there is always a M after the revision number and it isn't informative.

Anton Vodonosov (avodonosov) wrote :

Fare, I don't know what M means, I only saw it on release candidate, not normal releases.

CCL 1.9
--------------------------------------------------------------------------
                           asdf: ccl-1.9-f96-linux-x86
                     my variant: ccl-1.9-r15756-f96-linux-x86
 cl:lisp-implementation-version: Version 1.9-r15756 (LinuxX8632)
--------------------------------------------------------------------------

CCL 1.9 RC1
--------------------------------------------------------------------------
                           asdf: ccl-1.10-f96-linux-x86
                     my variant: ccl-1.10-rc1-r16188m-f96-linux-x86
 cl:lisp-implementation-version: Version 1.10-RC1-r16188M (LinuxX8632)
--------------------------------------------------------------------------

Faré (fahree) wrote :

The "M" means "modified". Since as part of its bootstrap, CCL currently overwrites the binary and image with newly compiled variants, which usually differ from those checked into SVN, most compiled checkouts will have an "M", even if the user did no modification, and so the "M" is uninformative. If the checked in version happens to coincide (because e.g. as part of a release, the latest image and binary were checked in, and the build process somehow makes them idempotent), then there is no "M", but this is not usually informative, except for people who might be manually patching a release — a very unlikely condition where the user probably knows what he's doing anyway.

pipping (pipping) wrote :

The general problem of describing the patch state of a lisp is very difficult.

Even though this bug is not about Allegro CL, I'll take a second to talk about it: On Allegro CL, for each platform you have a different set of patches with different numbers, that the user gets to install an arbitrary subset of. Not all patches are loaded at all times because they might affect parts of the system that are only loaded on-demand.

This suggests that ASDF cannot solve the general problem of describing the state of every supported lisp with the accuracy a human would expect (not too fine: if you make the string machine- and time-dependent, you're distinguishing states that you'd rather treat as identical; not too coarse, either: you should be able to tell if a patch has been applied to one of two lisps, both, or neither).

@avodonosov: cl:lisp-implementation-version provides you with the information you need. Are you sure that this is a problem which needs to be solved in ASDF and is not better tackled through a #+ccl in your code?

pipping (pipping) wrote :

I think the situation has changed quite a bit. More and more lisp implementations live on
git now and use `git describe --tags`-style information (possibly with processing) for their version, that we end up passing out via `(asdf:implementation-identifier)` [e.g. as](https://gitlab.common-lisp.net/asdf/asdf/snippets/24)

- cmu-snapshot-2017-04-42-g7f3040a7e__21b_unicode_-linux-x86
- ecl-16.1.3-f4feda8a-linux-x64
- mkcl-1.1.10.19-2dbfa99-linux-x64
- sbcl-1.3.21.149-9e3453954-linux-x64

CCL is now [on github](https://github.com/Clozure/ccl), too. I guess the only question here is wbo should do the work (probably them since they'll have to like the format in the end) if they haven't done it alreadt.

Changed in asdf:
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers