uiop:run-program broken on SBCL/Windows.

[SANO Masatoshi]

This is what I wrote to sbcl-devel.What's wrong with it is I can't see the result string.

$ ./run-sbcl.sh
(running SBCL from: .)
This is SBCL 1.2.14.21-55132b0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.

WARNING: the Windows port is fragile, particularly for multithreaded
code. Unfortunately, the development team currently lacks the time
and resources this platform demands.
* (require :uiop)

("UIOP" "uiop")
* *features*

(:OS-WINDOWS :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :ALIEN-CALLBACKS :ANSI-CL
 :ASH-RIGHT-VOPS :C-STACK-IS-CONTROL-STACK :COMMON-LISP :COMPARE-AND-SWAP-VOPS
 :COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :FLOAT-EQL-VOPS :FP-AND-PC-STANDARD-SAVE
 :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS :INTEGER-EQL-VOP
 :INTERLEAVED-RAW-SLOTS :LINKAGE-TABLE :LITTLE-ENDIAN :MEMORY-BARRIER-VOPS
 :MULTIPLY-HIGH-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-PUTWC
 :PACKAGE-LOCAL-NICKNAMES :PRECISE-ARG-COUNT-ERROR :RAW-INSTANCE-INIT-VOPS
 :SB-DOC :SB-DYNAMIC-CORE :SB-EVAL :SB-FUTEX :SB-LDB :SB-PACKAGE-LOCKS
 :SB-QSHOW :SB-SAFEPOINT :SB-SAFEPOINT-STRICTLY :SB-SIMD-PACK
 :SB-SOURCE-LOCATIONS :SB-TEST :SB-THREAD :SB-THRUPTION :SB-UNICODE :SB-WTIMER
 :SBCL :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-FIXED-OBJECTS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-VECTORS
 :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS
 :UNWIND-TO-FRAME-AND-CALL-VOP :WIN32 :X86-64)
* (uiop/run-program:run-program "c:/windows/system32/tree /?" :output :string)

"Graphically displays the folder structure of a drive or path.

TREE [drive:][path] [/F] [/A]

   /F Display the names of the files in each folder.
   /A Use ASCII instead of extended characters.

"
NIL
0
* (quit)

$ ./run-sbcl.sh
(running SBCL from: .)
This is SBCL 1.2.14.22-edf27f6, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.

WARNING: the Windows port is fragile, particularly for multithreaded
code. Unfortunately, the development team currently lacks the time
and resources this platform demands.
* (require :uiop)

("UIOP" "uiop")
* *features*

(:OS-WINDOWS :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :ALIEN-CALLBACKS :ANSI-CL
 :ASH-RIGHT-VOPS :C-STACK-IS-CONTROL-STACK :COMMON-LISP :COMPARE-AND-SWAP-VOPS
 :COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :FLOAT-EQL-VOPS :FP-AND-PC-STANDARD-SAVE
 :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS :INTEGER-EQL-VOP
 :INTERLEAVED-RAW-SLOTS :LINKAGE-TABLE :LITTLE-ENDIAN :MEMORY-BARRIER-VOPS
 :MULTIPLY-HIGH-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-PUTWC
 :PACKAGE-LOCAL-NICKNAMES :PRECISE-ARG-COUNT-ERROR :RAW-INSTANCE-INIT-VOPS
 :SB-DOC :SB-DYNAMIC-CORE :SB-EVAL :SB-FUTEX :SB-LDB :SB-PACKAGE-LOCKS
 :SB-QSHOW :SB-SAFEPOINT :SB-SAFEPOINT-STRICTLY :SB-SIMD-PACK
 :SB-SOURCE-LOCATIONS :SB-TEST :SB-THREAD :SB-THRUPTION :SB-UNICODE :SB-WTIMER
 :SBCL :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-FIXED-OBJECTS
 :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-VECTORS
 :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS
 :UNWIND-TO-FRAME-AND-CALL-VOP :WIN32 :X86-64)
* (uiop/run-program:run-program "c:/windows/system32/tree /?" :output :string)

""
NIL
0
* (quit)

[Robert P. Goldman (rpgoldman)]

What's the difference between these two cases?

There are different git variations of SBCL, right?

Does this happen on all windows binaries or only tree?

And why do we believe that this is an ASDF bug and not an SBCL bug?

I thought there was something about :force-shell implicated here, but I don't see it in either of these two cases.

Thanks

[Faré (fahree)]

The relevant difference is that SBCL upgraded its ASDF from 3.1.3 to 3.1.5, so that what broke this run-program invocation happened between 3.1.3 and 3.1.5 — and indeed, run-program was tweaked a lot at the time.

[Robert P. Goldman (rpgoldman)]

Thanks, Fare. I see: it was the ASDF-VERSION that was the important thing, not the SBCL version.

I will test this now.

[SANO Masatoshi (snmsts)]

Hi,

Thanks for testing.
I tried expression (uiop:run-program "tree /?" :output :string :force-shell
t)
with asdf "3.1.5.12" with downgrading sbcl's version and realized no sbcl
above sbcl 1.1.17 can get expected value.
Just I got "" always.

Thanks.

On Thu, Oct 1, 2015 at 2:55 AM, Robert P. Goldman <
<email address hidden>> wrote:

> Thanks, Fare. I see: it was the ASDF-VERSION that was the important
> thing, not the SBCL version.
>
> I will test this now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1501373
>
> Title:
> uiop:run-program broken on SBCL/Windows.
>
> Status in ASDF:
> New
>
> Bug description:
> This is what I wrote to sbcl-devel.What's wrong with it is I can't see
> the result string.
>
> $ ./run-sbcl.sh
> (running SBCL from: .)
> This is SBCL 1.2.14.21-55132b0, an implementation of ANSI Common Lisp.
> More information about SBCL is available at <http://www.sbcl.org/>.
>
> SBCL is free software, provided as is, with absolutely no warranty.
> It is mostly in the public domain; some portions are provided under
> BSD-style licenses. See the CREDITS and COPYING files in the
> distribution for more information.
>
> WARNING: the Windows port is fragile, particularly for multithreaded
> code. Unfortunately, the development team currently lacks the time
> and resources this platform demands.
> * (require :uiop)
>
> ("UIOP" "uiop")
> * *features*
>
> (:OS-WINDOWS :NON-BASE-CHARS-EXIST-P :ASDF-UNICODE :ALIEN-CALLBACKS
> :ANSI-CL
> :ASH-RIGHT-VOPS :C-STACK-IS-CONTROL-STACK :COMMON-LISP
> :COMPARE-AND-SWAP-VOPS
> :COMPLEX-FLOAT-VOPS :CYCLE-COUNTER :FLOAT-EQL-VOPS
> :FP-AND-PC-STANDARD-SAVE
> :GENCGC :IEEE-FLOATING-POINT :INLINE-CONSTANTS :INTEGER-EQL-VOP
> :INTERLEAVED-RAW-SLOTS :LINKAGE-TABLE :LITTLE-ENDIAN
> :MEMORY-BARRIER-VOPS
> :MULTIPLY-HIGH-VOPS :OS-PROVIDES-DLOPEN :OS-PROVIDES-PUTWC
> :PACKAGE-LOCAL-NICKNAMES :PRECISE-ARG-COUNT-ERROR
> :RAW-INSTANCE-INIT-VOPS
> :SB-DOC :SB-DYNAMIC-CORE :SB-EVAL :SB-FUTEX :SB-LDB :SB-PACKAGE-LOCKS
> :SB-QSHOW :SB-SAFEPOINT :SB-SAFEPOINT-STRICTLY :SB-SIMD-PACK
> :SB-SOURCE-LOCATIONS :SB-TEST :SB-THREAD :SB-THRUPTION :SB-UNICODE
> :SB-WTIMER
> :SBCL :STACK-ALLOCATABLE-CLOSURES :STACK-ALLOCATABLE-FIXED-OBJECTS
> :STACK-ALLOCATABLE-LISTS :STACK-ALLOCATABLE-VECTORS
> :STACK-GROWS-DOWNWARD-NOT-UPWARD :SYMBOL-INFO-VOPS
> :UNWIND-TO-FRAME-AND-CALL-VOP :WIN32 :X86-64)
> * (uiop/run-program:run-program "c:/windows/system32/tree /?" :output
> :string)
>
> "Graphically displays the folder structure of a drive or path.
>
> TREE [drive:][path] [/F] [/A]
>
> /F Display the names of the files in each folder.
> /A Use ASCII instead of extended characters.
>
> "
> NIL
> 0
> * (quit)
>
> $ ./run-sbcl.sh
> (running SBCL from: .)
> This is SBCL 1.2.14.22-edf27f6, an implementation of ANSI Common Lisp.
> More information about SBCL is available at <http://www.sbcl.org/>.
>
> SBCL is free software, provided as is, with absolutely no warranty.
> It is mostly in the public domain; some portions are provided under
> BSD...

[Robert P. Goldman (rpgoldman)]

I am adding the output from tracing this on the newest ASDF. See file attachment.

[Robert P. Goldman (rpgoldman)]

I am adding the output from tracing this on the newest ASDF. See file attachment.

[Faré (fahree)]

Can you grab ASDF 3.1.3 from e.g.
https://common-lisp.net/project/asdf/archives/asdf-3.1.3.lisp
and trace again?

This trace doesn't show anything suspect, besides perhaps the fact that we're not using < > for redirection even though we are kind of going through the shell.

[Faré (fahree)]

<minion> resttime, memo from Fare: to help with uiop:run-program on sbcl/windows, see https://bugs.launchpad.net/asdf/+bug/1501373
<resttime> Ah Fare: I think that's a specific bug in general which I encountered with CFFI
<Fare> GCL is not super maintained
<nyef> resttime: And now that you know, you can ignore it deliberately, like almost everyone else does.
<Fare> ASDF support is lacking
<Fare> and the maintainer is not responding to my mail about it
<resttime> Basically you can't redirect the run-program output in windows
<Fare> once every so many months for a few years now
<resttime> However you can open a direct stream and read that output
<resttime> Let me go find the patch for cffi that
<resttime> Fare: https://github.com/cffi/cffi/commit/a9ec33b2aac3130cc4cc6fcae75bd0fd13718b82
<resttime> Actually wait different SBCL versions and it works hmmm...
<resttime> Hmmmm, on second thought it might be related
<Fare> resttime: this is not helping locate the bug in uiop
<Fare> and/or fix the limitations in sbcl
<resttime> Fare: Okay yeah, grabbed asdf 3.1.3 downgraded and it works
<resttime> SBCL 1.2.15
<resttime> Fare: okay I found the bug
<resttime> Line 897: https://gitlab.common-lisp.net/asdf/asdf/blob/master/uiop/run-program.lisp
<resttime> (when (stringp command)
<resttime> (unless force-shell-suppliedp
<resttime> (setf force-shell t)))
<resttime> comment that out and it should work I believe
<resttime> Fare: Hmm I guess more specifically if FORCE-SHELL is T then there is no output
<resttime> (uiop/run-program:run-program "c:/windows/system32/tree /?" :output :string :force-shell nil) works
<resttime> If the keyword isn't supplied then FORCE-SHELL is set to T by default
<resttime> https://gitlab.common-lisp.net/asdf/asdf/commit/9d86792973e2bf11128b74fb8fd616ae0d9cfd6c
<resttime> This commit
* stardiviner (~stardivin@unaffiliated/stardiviner) has joined #lisp
<resttime> Unfortunately, I think I've only found a superficial work around so far...
<resttime> Err also to clarify the command specifically works on SBCL 1.2.15 when keyword FORCE-SHELL is NIL
<Fare> what does force-shell do?
<Fare> I can guess what it does.
<Fare> and why that wouldn't work with SBCL
<Fare> resttime, ok, I pushed a 3.1.5.20 -- can you test?
<Fare> resttime: thanks a lot for tracking down the issue
<resttime> Fare: From what I can figure force-shell tries to do things without a stream to a process
<resttime> also no problem, let me just test the push now...
<resttime> Fare: yup working now
<resttime> Err * without a stream the command's process
<resttime> *to
<Fare> ?
<resttime> Which is %system and %use-system
<Fare> I didn't understand your remark
<Fare> is it working or not?
<resttime> oh I was referring to what I was trying to figure about what was broken
<resttime> yes it works

[Faré (fahree)]

A real fix would entail fixing SBCL to provide a raw interface to CMD.EXE without extra quotes, like CCL does.