Allow direct run-program without quoting on Windows

Bug #1503496 reported by Faré on 2015-10-07
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
SBCL
Undecided
Unassigned

Bug Description

SBCL fails to offer an interface to CreateProcess without quoting arguments, which is essential to issue Windows CMD.EXE commands and pipes. Cygwin's bash cannot be portably relied upon on Windows, and SBCL thus makes it hard to portably orchestrate processes from Lisp.

As an example of interface that works, see what Allegro, LispWorks or CCL provide. CCL has the advantage of being free software, so you can look how they do it inside, if needs be. See CCL bug 858 about this feature: http://trac.clozure.com/ccl/ticket/858

Allegro and LispWorks offer a raw command line as a single string. CCL requires the user to separate the name of the program from the rest of the command string (though maybe it also accepts NIL to pass NULL to the underlying Windows system call).

https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

Faré (fahree) wrote :

Here is a minimal patch that allows for direct access to the arguments to CreateProcess on win32.

It allows SBCL to pass all asdf tests, in our run-program-windows branch:
https://gitlab.common-lisp.net/asdf/asdf/commits/run-program-windows

Faré (fahree) wrote :

When I tried the other day, CMD commands that required proper quoting, such as findstr "^", could not be issued with the current quoting regime.

Stas Boukarev (stassats) wrote :

You have to be more specific.

Faré (fahree) wrote :

Let's take a trivial Windows scripting example, the equivalent of cat < in > out 2> err,
where the rough equivalent of cat is the internal command findstr "^"
where the caret must be properly quoted.

;; With my patch, you can run this command, that works:
(sb-ext:RUN-PROGRAM "cmd" "/c findstr \"^\"" :INPUT #P"c:/cygwin64/home/fare/in"
                  :OUTPUT #P"c:/cygwin64/home/fare/out" :ERROR
                  #P"c:/cygwin64/home/fare/err" :WAIT T :EXTERNAL-FORMAT :UTF-8 :IF-INPUT-DOES-NOT-EXIST
                  :ERROR :IF-OUTPUT-EXISTS :OVERWRITE :IF-ERROR-EXISTS :OVERWRITE :SEARCH T)
;; It calls:
(SB-WIN32::MSWIN-SPAWN "cmd" "cmd /c findstr \"^\"" 712 732 804 T #.(SB-SYS:INT-SAP #X00000000) T NIL)

;; Without my patch, I can't make it work.

;; This trivial attempt fails with exit code 2, because it doesn't quote enough:
(sb-ext:RUN-PROGRAM "cmd" '("/c" "findstr" "^") :INPUT #P"c:/cygwin64/home/fare/in"
                  :OUTPUT #P"c:/cygwin64/home/fare/out" :ERROR
                  #P"c:/cygwin64/home/fare/err" :WAIT T :EXTERNAL-FORMAT :UTF-8 :IF-INPUT-DOES-NOT-EXIST
                  :ERROR :IF-OUTPUT-EXISTS :OVERWRITE :IF-ERROR-EXISTS :OVERWRITE :SEARCH T)
(SB-WIN32::MSWIN-SPAWN "cmd" "cmd /c findstr ^" 852 796 696 T #.(SB-SYS:INT-SAP #X00000000) T NIL)

;; This fails with exit code 1, because it quotes too much:
(sb-ext:RUN-PROGRAM "cmd" '("/c" "findstr" "\"^\"") :INPUT #P"c:/cygwin64/home/fare/in"
                  :OUTPUT #P"c:/cygwin64/home/fare/out" :ERROR
                  #P"c:/cygwin64/home/fare/err" :WAIT T :EXTERNAL-FORMAT :UTF-8 :IF-INPUT-DOES-NOT-EXIST
                  :ERROR :IF-OUTPUT-EXISTS :OVERWRITE :IF-ERROR-EXISTS :OVERWRITE :SEARCH T)
(SB-WIN32::MSWIN-SPAWN "cmd" "cmd /c findstr \"\\\"^\\\"\"" 808 796 780 T #.(SB-SYS:INT-SAP #X00000000) T NIL)

;; Trying to use multiple nested cmd /c to achieve unquoting doesn't obviously work either (exit 1):
(sb-ext:RUN-PROGRAM "cmd" '("/c" "cmd" "/c" "findstr \"^\"") :INPUT #P"c:/cygwin64/home/fare/in"
                  :OUTPUT #P"c:/cygwin64/home/fare/out" :ERROR
                  #P"c:/cygwin64/home/fare/err" :WAIT T :EXTERNAL-FORMAT :UTF-8 :IF-INPUT-DOES-NOT-EXIST
                  :ERROR :IF-OUTPUT-EXISTS :OVERWRITE :IF-ERROR-EXISTS :OVERWRITE :SEARCH T)
(SB-WIN32::MSWIN-SPAWN "cmd" "cmd /c cmd /c \"findstr \\\"^\\\"\"" 808 804 892 T #.(SB-SYS:INT-SAP #X00000000) T NIL)

;; In this particular case, I can finesse cmd because the character to be quoted was the ^, which is a shell escape character like \ in Unix shell:
(sb-ext:RUN-PROGRAM "cmd" '("/c" "findstr" "^^") :INPUT #P"c:/cygwin64/home/fare/in"
                  :OUTPUT #P"c:/cygwin64/home/fare/out" :ERROR
                  #P"c:/cygwin64/home/fare/err" :WAIT T :EXTERNAL-FORMAT :UTF-8 :IF-INPUT-DOES-NOT-EXIST
                  :ERROR :IF-OUTPUT-EXISTS :OVERWRITE :IF-ERROR-EXISTS :OVERWRITE :SEARCH T)
(SB-WIN32::MSWIN-SPAWN "cmd" "cmd /c findstr ^^" 760 1108 968 T #.(SB-SYS:INT-SAP #X00000000) T NIL)

;; But what if you want to run an arbitrary command, wit carets and quotes, etc.? Maybe someone can write a clever quoting algorithm just to overcome this SBCL hurdle. Or SBCL can get a pass-thru mode.

Stas Boukarev (stassats) wrote :

That makes it clearer.

Faré (fahree) wrote :

Is there any chance of getting this into SBCL 1.3.13?

How may I amend the proposed patch to make it acceptable?

Jan Moringen (scymtym) wrote :

> How may I amend the proposed patch to make it acceptable?

I cannot promise that addressing these issues will get the patch accepted, but from previous discussions, I remember the following two points

1) Looking at the patch, rpg, stassats and me wondered why the added docstring bit was conditionalized but the added code wasn't.

2) If possible when relying only on guaranteed-to-be-available windows programs, adding a test that exercises the new behavior would be good.

Thanks for your patience.

Stas Boukarev (stassats) wrote :

I added a new option, :escape-arguments.

In fcae0fd0e0c8247cb4d1cb01a0e474b93f654ab2

Changed in sbcl:
status: New → Fix Committed
Changed in sbcl:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers