xsltproc is not able to handel \ in paths when creating BOM lists

[Stefan Olsson]

Dear Sir or Madame

Platform: Windows 7
kiCad version: (2015-07-04 BZR 5884, Git c1bbf3e)
xsltproc version: Using libxml 20902, libxslt 10128 and libexslt 817

Problem:
Command line xsltproc -o "%O.csv" "%P/bom_with_title_block_2_csv.xsl" "%I" will fail if running on Windows 7

Cause:
The above line is expanded into

xsltproc -o "G:\Project\24\B Projekt\KraftKort.csv" "G:\Project\24\B Projekt\KraftKort/bom_with_title_block_2_csv.xsl" "G:\Project\24\B Projekt\KraftKort\KraftKort.xml"

This line is not working for xsltproc becosue it can not handle the /, \ issue between running on Microsoft Windows/unix/linux platforms

That is, the windows version of xsltproc can only handle / on a Microsoft Windows platform.

(if the above line is replaced with xsltproc -o "G:/Project/24/B Projekt/KraftKort.csv" "G:/Project/24/B Projekt/KraftKort/bom_with_title_block_2_csv.xsl" "G:/Project/24/B Projekt/KraftKort/KraftKort.xml" it will work)

Suggested solution:
It is suggest that KiCad first expand the command line (as it do today) and then, if running on
Microsoft Windows: Replace all \ in the string with /
Unix/Linux: Replace all / in the string with \

In this way the KiCad BOM generation will work on both Microsoft Windows and Unix/Linux platforms regardless on which platform the plugins was originally written on or will be executed on.

Extra:
I will report the issue to xsltproc project as well.

PS
I dont know with directory seperator character OSX is using but it is originaly a Unix distribution
DS

Regards
/Stefan

[Nick Østergaard (nickoe)]

Eeh?!? "Unix/Linux: Replace all / in the string with \" why do you want to do that?

[Stefan Olsson (stefan-olsson)]

Hi

Unless Unix/Linux can accept both / and \ as directory seperator character, a line like this
(assume that the command line was created in KiCad running on Microsoft windows and then moved/used on
KiCad running under Unix/Linux)

xsltproc -o "../../myown/bom.csv" ...............

will be expanded into

xsltproc -o ../../myown/bom.csv

and I assume / is not understod by a Unix/Liunx shell (which i assume kiCad is using to execute the comamnd line)

therefore my suggestion include the use case in where someone have written a relative path on the command line
in KiCad on Microsoft windows and then it is used on Unix/Linux.

/Stefan

[Nick Østergaard (nickoe)]

The forward slash is the only accepted path separator on unix. On windows you can use both.

[Stefan Olsson (stefan-olsson)]

Mmm, I have mixed it up in my post, let me check it once more to get it right.
(thanks to Tim Paterson we are living with this mess)

You are right, it seems that it is xsltproc alone that cant handle the correct path seperator
for the platfrom it is executing on and KiCad is doing the right thing.

[Stefan Olsson (stefan-olsson)]

Well I have tested around a little and found the following

On Microsoft windows 7

the following works

dir c:\temp\bilder
dir "c:\temp\bilder"
dir "c:/temp/bilder"

but not

dir c:/temp/bilder

usage of / on microsoft windows require "" so dir "c:/temp/bilder"
(but to make it even worse)

The following works
dir c:\temp\bilder\Maria.jpg
dir "c:\temp\bilder\Maria.jpg"

but not
dir "c:/temp/bilder/Maria.jpg"

Microsoft windows require that the last path seperator is\, so the following works
dir "c:/temp/bilder\Maria.jpg"

which means that the "both /\ can be used on Microsoft windows" is a truth with modification

This means in turn that if the command line look like this
xsltproc -o "%O.csv" "%P/bom_with_title_block_2_csv.xsl" "%I"

The last / will not be "translated" wich means that on Microsoft windows you will get a path that have mixed /\

So my suggestion to change is still recomended (with the mixup i did fixed as you pointed out)

Microsoft Windows: Replace all / in the string with \
Unix/Linux: Replace all \ in the string with /

PS
I discoved also that xsltproc is not handling spaces in paths (regardless of usage of \ or /) very well.
I am looking into that as well but is not a KiCad issue.
DS

[Nick Østergaard (nickoe)]

So as I read it, using quoted forward slashes should work cross platform. Why not just suggest that then? Why do we need to convert to backward slashes on windows if it works with forward slashes in quotes? In any case it is always wise to quote path strings, to avoid using escape characters dor spaces.

[Stefan Olsson (stefan-olsson)]

In order to get forward slash to work under microsoft windows the path must be quoted

BUT

the last path seperator, when pointing to a file, must be backslask, even if the whoel path expression is quoted.

Example
xsltproc -o "%O.csv" "%P/bom_with_title_block_2_csv.xsl" "%I"

will not work under microsoft windows becosue the last path seperator in "%P/bom_with_title_block_2_csv.xsl" is a
forward slash and that will not be accepted by the microsoft windows shell,

you can try it by doing the following (assume you have a mypicture.jpg in c:\temp)

dir "c:/temp/mypicture.jpg"

this will generate an error (file not found),

but the following will work

dir "c:/temp\mypicture.jpg"

It does not matter how deep the path is, the last path seperator need to be microsoft windows native path seperator

[Wayne Stambaugh (stambaughw)]

Are you running xsltproc directly from the command line or as an Eeschema netlist plugin?

[Stefan Olsson (stefan-olsson)]

It is executed as an Eeschema netlist plugin

I digged into this even further and there are two problems

1)
The problem described above with the / at the end in a "" closed path under microsoft windows
(where i suggest that KiCad should replace all / with \ in the resulting path under microsoft windows
before calling the CMD shell for executing)
Is really a microsoft problem and is not a "true KiCad error", it is more of making KiCad more "error tolerant"

2) (not reallly a KiCad problem)
Problem with xsltproc itself and handling of path to the -o arguments
xsltproc have problems with the -o argument (under microsoft windows), see my report here
https://bugzilla.gnome.org/show_bug.cgi?id=752219
which might be an duplicate of
https://bugzilla.gnome.org/show_bug.cgi?id=750365

xsltproc does not handle absolute path under microsoft windows for the -o argument correctly
(the other two paths given to xsltproc is working as expected)

You can get around it by calling a .bat file from the Eeschema netlist plugin
using known intermediate files when calling for xsltproc and then copy it back to "%O.csv"

"Fixing problem" 1 would mean that KiCad will be less sensitive if the project is created on /Unix/Linux/MSWindows and
if it is shared between the platforms (however the usage native microsoft .bat files destroyes that possibility anyway)

[Wayne Stambaugh (stambaughw)]

I just tested and did not have any issues using Eeschema version:

Application: kicad
Version: (2015-08-10 BZR 6083)-product release build
wxWidgets: Version 3.0.2 (debug,wchar_t,compiler with C++ ABI 1009,GCC 5.2.0,wx containers,compatible with 2.8)
Platform: Windows 7 (build 7601, Service Pack 1), 64-bit edition, 64 bit, Little endian, wxMSW
Boost version: 1.57.0
         USE_WX_GRAPHICS_CONTEXT=OFF
         USE_WX_OVERLAY=OFF
         KICAD_SCRIPTING=ON
         KICAD_SCRIPTING_MODULES=ON
         KICAD_SCRIPTING_WXPYTHON=ON
         USE_FP_LIB_TABLE=HARD_CODED_ON
         BUILD_GITHUB_PLUGIN=ON

and xsltproc.exe version:

Using libxml 20902, libxslt 10128 and libexslt 817
xsltproc was compiled against libxml 20902, libxslt 10128 and libexslt 817
libxslt 10128 was compiled against libxml 20902
libexslt 817 was compiled against libxml 20902

Both were compiled from source using msys2 and mingw64 on W7 pro.

[jean-pierre charras (jp-charras)]

I am thinking some versions of xsltproc.exe do not accept \ as separator
On my msys2 install (same version as your, but 32 bits) \ in command line does not work when inside the filename following the option -o .

My previous xsltproc.exe worked fine with \ in command line...

Since rev 6013, all \ are replaced by / in command line, which works on all platforms.

[Wayne Stambaugh (stambaughw)]

Does this mean that this bug can be tagged as fix committed?

On 8/10/2015 3:09 PM, jean-pierre charras wrote:
> I am thinking some versions of xsltproc.exe do not accept \ as separator
> On my msys2 install (same version as your, but 32 bits) \ in command line does not work when inside the filename following the option -o .
>
> My previous xsltproc.exe worked fine with \ in command line...
>
> Since rev 6013, all \ are replaced by / in command line, which works on
> all platforms.
>

[jean-pierre charras (jp-charras)]

Because since rev 6013 (2015, July 29), no issue was found, I tag it as fix committed.

[Beuf (spamdebeuf)]

I ran into a similar issue and Stefan Olsson (stefan-olsson) was right (remark #5). This issue was discovered on windows 7.

I got an error '11' message

Run command:
xsltproc -o "C:/AR-W7/ED-Projects/Designs/Txx Test boards/TG1 General protoboard/TG1.csv" "C:/Program Files/KiCad/bin/scripting/plugins/bom2csv.xsl" "C:/AR-W7/ED-Projects/Designs/Txx Test boards/TG1 General protoboard/TG1.xml"

Command error. Return code 11

Error messages:
I/O error : Invalid argument
I/O error : Invalid argument

I could trace it back to spaces in the output file name (the .csv file). Spaces in the other paths are not an issue. This is of course an xsltproc bug and not something to fix in KiCad.

[Beuf (spamdebeuf)]

A simple work around is to change the output file into a fixed file instead of a project file (this is easier then changing my folder structure). In this case all works fine:

Run command:
xsltproc -o "C:/AR-W7/ED-Projects/Designs/BOM.csv" "C:/Program Files/KiCad/bin/scripting/plugins/bom2csv.xsl" "C:/AR-W7/ED-Projects/Designs/Txx Test boards/TG1 General protoboard/TG1.xml"

Success

[Nick Østergaard (nickoe)]

@Beuf, is it not possible to escape the spaces in the file name with for example \ (backslash)?

[Beuf (spamdebeuf)]

@Nick

KiCad already generates the path between quotes which should be fine. It is for the other parameters, not for the output file.

I tried replacing the spaces with %20 with and without quotes. This didn't work as well.

I tried directing the stdout to a file and skipping the -o option:

xsltproc "C:/Program Files/KiCad/bin/scripting/plugins/bom2csv.xsl" "C:/AR-W7/ED-Projects/Designs/Txx Test boards/TG1 General protoboard/TG1.xml" > "C:/AR-W7/ED-Projects/Designs/Txx Test boards/TG1 General protoboard/TG1.csv"

This works in command line mode but not in the KiCad BOM generation interface as it captures stdout for error reporting.

[Beuf (spamdebeuf)]

Similar thread on another forum. Other work around is to use saxon xlst processor.

https://forum.kicad.info/t/bom-i-o-error-invalid-argument/1369