translate-pathname: break with file version: :NEWEST vs. NIL

[kr]

as a disclaimer, this is the 1st time i'm filing a bug on launchpad,
and while i think that this is probably a new bug (given over 500 open
bugs), i don't generally know much about logical pathnames.

i was investigating another, unrelated bug in the pretty printer,
regarding indenting and ~i not working in my hands in sbcl-1.3.1

eventually, i wondered whether some test suite might have excercised
these issues. so i downloaded the latest snapshot of ansi-test, as
described at https://common-lisp.net/project/ansi-test/

the installation instructions state running the doit.lsp file, which
immediately broke in sbcl-1.3.1 , before running any actual tests, and
this is the subject of this bug report.

first, the error msg. and partial stack backtrace:

===========
...
While evaluating the form starting at line 28, column 0
  of #P"/opt/test/ansi-test/gclload1.lsp":
While evaluating the form starting at line 18, column 0
  of #P"/opt/test/ansi-test/doit.lsp":

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10039AC803}>:
  Pathname components from SOURCE and FROM args to TRANSLATE-PATHNAME
did not match:
  :NEWEST NIL

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY ] Retry EVAL of current toplevel form.
  1: [CONTINUE] Ignore error and continue loading file "/opt/test/ansi-test/gclload1.lsp".
  2: [ABORT ] Abort loading file "/opt/test/ansi-test/gclload1.lsp".
  3: Retry EVAL of current toplevel form.
  4: Ignore error and continue loading file "/opt/test/ansi-test/doit.lsp".
  5: Abort loading file "/opt/test/ansi-test/doit.lsp".
  6: Exit debugger, returning to top level.

(SB-IMPL::DIDNT-MATCH-ERROR :NEWEST NIL)
0] :back

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10039AC803}>
0: (SB-IMPL::DIDNT-MATCH-ERROR :NEWEST NIL)
1: (SB-IMPL::TRANSLATE-COMPONENT :NEWEST NIL :NEWEST T)
2: (TRANSLATE-PATHNAME #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST" #P"ANSI-TESTS:AUX;*.*" #P"/opt/test/ansi-test/auxiliary/")
3: (TRANSLATE-LOGICAL-PATHNAME #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST")
4: (SB-IMPL::QUERY-FILE-SYSTEM #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST" :WRITE-DATE T)
5: (FILE-WRITE-DATE #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST")
6: (COMPILE-AND-LOAD "ANSI-TESTS:AUX;ansi-aux-macros.lsp" :FORCE NIL)
7: ((LAMBDA NIL :IN "/opt/test/ansi-test/gclload1.lsp"))
8: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET (*LOAD-VERBOSE* *LOAD-PRINT* *COMPILE-VERBOSE* *COMPILE-PRINT*) (LOAD "rt-package.lsp") (COMPILE-AND-LOAD "rt.lsp") (LOAD "cl-test-package.lsp") (IN-PACKAGE :CL-TEST) (COMPILE-AND-LOAD "ANSI-TESTS:AUX;ansi-aux-macros.lsp") (COMMON-LISP:HANDLER-BIND ((SB-EXT:CODE-DELETION-NOTE (FUNCTION MUFFLE-WARNING))) (LOAD "universe.lsp")) (COMPILE-AND-LOAD "ANSI-TESTS:AUX;random-aux.lsp") (COMPILE-AND-LOAD "ANSI-TESTS:AUX;ansi-aux.lsp") (LOAD "cl-symbol-names.lsp") (LOAD "notes.lsp")) #<NULL-LEXENV>)
...
===========

second, i started tracing some internals to hopefully get a better understanding of this.
this was from the same session, after getting back to toplevel.

===========
COMMON-LISP-USER > (probe-file "ANSI-TESTS:AUX;ansi-aux-macros.lsp")
  0: (TRANSLATE-PATHNAME #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST"
                         #P"ANSI-TESTS:AUX;*.*"
                         #P"/opt/test/ansi-test/auxiliary/")
    1: (SB-IMPL::TRANSLATE-COMPONENT :UNSPECIFIC :UNSPECIFIC NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned :UNSPECIFIC
    1: (SB-IMPL::TRANSLATE-COMPONENT "ANSI-AUX-MACROS" :WILD NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned "ansi-aux-macros"
    1: (SB-IMPL::TRANSLATE-COMPONENT "LSP" :WILD NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned "lsp"
    1: (SB-IMPL::TRANSLATE-COMPONENT :NEWEST NIL :NEWEST T)

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10039AC733}>:
  Pathname components from SOURCE and FROM args to TRANSLATE-PATHNAME
did not match:
  :NEWEST NIL

===========

third, i tried to fix it.

the code for translate-pathname seemed to call translate-component on
the version component as well, even though translate-component has an
explicit comment stating that the version is supposed to be an
exception too. i ended up creating a patch that i will attach, which
creates a new function called translate-versions , which then is
called from translate-pathname. while i don't claim to understand the
microscopic details of so-called "logical" pathnames and what
conventions ought to be used for version comparisons, given that
probably nobody has been using that after the demise of symbolics, i
think it is a good idea to sequester that potential problem into its
own function, so it can be more easily traced and tinkered with,
should the need arise again. currently, translate-versions is
primarily a copy of translate-component , but which also deals with
:newest in a simple (simplistic ?) way.

after this fix, the break i ran into no longer occurs:

===========
COMMON-LISP-USER > (probe-file "ANSI-TESTS:AUX;ansi-aux-macros.lsp")
  0: (TRANSLATE-PATHNAME #P"ANSI-TESTS:AUX;ANSI-AUX-MACROS.LSP.NEWEST"
                         #P"ANSI-TESTS:AUX;*.*"
                         #P"/opt/test/ansi-test/auxiliary/")
    1: (SB-IMPL::TRANSLATE-COMPONENT :UNSPECIFIC :UNSPECIFIC NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned :UNSPECIFIC
    1: (SB-IMPL::TRANSLATE-COMPONENT "ANSI-AUX-MACROS" :WILD NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned "ansi-aux-macros"
    1: (SB-IMPL::TRANSLATE-COMPONENT "LSP" :WILD NIL T)
    1: SB-IMPL::TRANSLATE-COMPONENT returned "lsp"
    1: (SB-IMPL::TRANSLATE-VERSIONS :NEWEST NIL :NEWEST T)
    1: SB-IMPL::TRANSLATE-VERSIONS returned :NEWEST
  0: TRANSLATE-PATHNAME returned
       #P"/opt/test/ansi-test/auxiliary/ansi-aux-macros.lsp"
#P"/opt/test/ansi-test/auxiliary/ansi-aux-macros.lsp"
COMMON-LISP-USER >
===========

furthermore, i rebuilt sbcl-1.3.1 with my code fix, and it built fine.
i ran the sbcl test suite before and after, and the the final result
looks the same. i don't think my patch thus causes a problem, even
though it maybe is not a final and perfect fix.

but now, sbcl users can run ansi-test again, and see all the even more
serious problems that are revealed... :-(

uname -a
Linux firebird 3.2.0-69-generic #103-Ubuntu SMP Tue Sep 2 05:02:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

* *features*

(:64-BIT :64-BIT-REGISTERS :ALIEN-CALLBACKS :ANSI-CL :ASH-RIGHT-VOPS
 :C-STACK-IS-CONTROL-STACK :COMMON-LISP :COMPARE-AND-SWAP-VOPS
 :COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :ELF :FLOAT-EQL-VOPS
 :FP-AND-PC-STANDARD-SAVE :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS
 :INTEGER-EQL-VOP :INTERLEAVED-RAW-SLOTS :LARGEFILE :LINKAGE-TABLE :LINUX
 :LITTLE-ENDIAN :MEMORY-BARRIER-VOPS :MULTIPLY-HIGH-VOPS :OS-PROVIDES-BLKSIZE-T
 :OS-PROVIDES-DLADDR :OS-PROVIDES-DLOPEN :OS-PROVIDES-GETPROTOBY-R
 :OS-PROVIDES-POLL :OS-PROVIDES-PUTWC :OS-PROVIDES-SUSECONDS-T
 :PACKAGE-LOCAL-NICKNAMES :PRECISE-ARG-COUNT-ERROR :RAW-INSTANCE-INIT-VOPS
 :SB-DOC :SB-EVAL :SB-FUTEX :SB-LDB :SB-PACKAGE-LOCKS :SB-SIMD-PACK
 :SB-SOURCE-LOCATIONS :SB-TEST :SB-THREAD :SB-THREAD :SB-UNICODE
 :SB-XREF-FOR-INTERNALS :SBCL :STACK-ALLOCATABLE-CLOSURES
 :STACK-ALLOCATABLE-FIXED-OBJECTS :STACK-ALLOCATABLE-LISTS
 :STACK-ALLOCATABLE-VECTORS :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS
 :UNIX :UNWIND-TO-FRAME-AND-CALL-VOP :X86-64)

[Richard M Kreuter (kreuter)]

Here's a simplified reproduction:

* (translate-pathname (make-pathname :host "SYS" :version :newest)
                      (make-pathname :host "SYS" :version nil)
                      (make-pathname :defaults "/" :version :newest))

This is only one way that TRANSLATE-PATHNAME sometimes disagrees with PATHNAME-MATCH-P. More examples:

* (pathname-match-p "foo" "") => T, but

* (translate-pathname "foo" "" "*-bar") ; errors

There are couple things contributing to this peculiar bug:

(1) SB-IMPL::COMPONENT-MATCH doesn't handle NIL (which is okay, as it's a sub-primitive that's also used by SB-IMPL::DIRECTORY-COMPONENTS-MATCH), so some of SB-IMPL::COMPONENT-MATCH's callers have to handle NIL components explicitly (PATHNAME-MATCH-P does, SB-IMPL::TRANSLATE-COMPONENT doesn't); and

(2) SB-IMPL::TRANSLATE-COMPONENT dispatches on the type of TO, then FROM, then SOURCE, and when things can fall through that dispatch, DIDNT-MATCH gets signaled. It's somewhat tricky to reason through that dispatching's totality.

Minimal change attached. In case anyone cares to make this better, IMO TRANSLATE-PATHNAME should call PATHNAME-MATCH-P before trying to do any translating, and SB-IMPL::TRANSLATE-COMPONENT could then signal a bug when its dispatching fails.