EpicsHostArch script does not detect recent MSYS2/MinGW setups on Windows

[Ralph Lange]

On MSYS2/MinGW installations, the EpicsHostArch script does not recognize the host platform correctly.

64bit:
$ ./startup/EpicsHostArch
lib/perl/EpicsHostArch.pl: Architecture 'x86_64-msys-thread-multi' not recognized

32bit:
$ ./startup/EpicsHostArch
lib/perl/EpicsHostArch.pl: Architecture 'x86_64-msys-thread-multi' not recognized

Which is not surprising, as the underlying MSYS2 subsystem is the same, and the difference is in the environment settings that point to the appropriate toolchain.

64bit:
$ echo $PATH
/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
$ printenv | grep MSYSTEM
MSYSTEM_CARCH=x86_64
MSYSTEM_CHOST=x86_64-w64-mingw32
MSYSTEM_PREFIX=/mingw64
MSYSTEM=MINGW64

32bit:
$ echo $PATH
/mingw32/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
$ printenv | grep MSYSTEM
MSYSTEM_CARCH=i686
MSYSTEM_CHOST=i686-w64-mingw32
MSYSTEM_PREFIX=/mingw32
MSYSTEM=MINGW32

NB: I am impressed by the clarity and simplicity of MSYS2 / MinGW

[Andrew Johnson (anj)]

How are you distinguishing between 32- and 64-bit? Your CPU can't really be 32-bit or a Perl built for x86_64 wouldn't run on it. Does this same Perl version come with both, or neither?

For existing targets we have taken the Perl architecture as indicative of what the host actually runs on. There may be strange effects (especially in modules/ca/src/perl) if it doesn't match.

The output from 'perl -V' would be helpful, at least it will tell me about the 64-bit system.

[Ralph Lange (ralph-lange)]

Seems I wasn't clear enough:

This is *one* MSYS2 system (on a 64bit machine), with the packages for both the toolchains for mingw32 and mingw64 installed.
The only difference between using it to compile MinGW 64bit or 32bit executables is the environment setting. (In the similar fashion as on the Windows command shell, where the difference between using VisualStudio 32bit and 64bit compilers is the environment setting achieved by running vsvarsall.bat.)

Perl is running on MSYS2.

$ perl -V
Summary of my perl5 (revision 5 version 28 subversion 2) configuration:

  Platform:
    osname=msys
    osvers=3.0.6(0.33853)
    archname=x86_64-msys-thread-multi
    uname='msys_nt-6.1 warlock 3.0.6(0.33853) 2019-04-15 06:44 x86_64 msys '
    config_args='-des -Dusethreads -Doptimize=-march=x86-64 -mtune=generic -O2 -pipe -Dprefix=/usr -Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl -Darchlib=/usr/lib/perl5/core_perl -Dsitelib=/usr/share/perl5/site_perl -Dsitearch=/usr/lib/perl5/site_perl -Dvendorlib=/usr/share/perl5/vendor_perl -Dvendorarch=/usr/lib/perl5/vendor_perl -Dscriptdir=/usr/bin/core_perl -Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl -Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl -Darchname=-msys-threads -Dmyarchname=-msys -Dlibperl=msys-perl5_28.dll -Dcc=gcc -Dld=g++ -Accflags=-march=x86-64 -mtune=generic -O2 -pipe -fwrapv'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong'
    optimize='-march=x86-64 -mtune=generic -O2 -pipe'
    cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -fno-strict-aliasing -fstack-protector-strong'
    ccversion=''
    gccversion='7.4.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'
    libpth=/usr/lib
    libs=-lpthread -lgdbm -ldb -ldl -lcrypt -lgdbm_compat
    perllibs=-lpthread -ldl -lcrypt
    libc=/usr/lib/libmsys-2.0.a
    so=dll
    useshrplib=true
    libperl=msys-perl5_28.dll
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' --shared -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'

Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
...

[Andrew Johnson (anj)]

Okay thanks. The EpicsHostArch.pl script was designed for auto-detection of "what kind of host OS am I running on" which it does by asking Perl. To build for any other architecture that I know my machine is also capable of building for I would expect to have to manually set the EPICS_HOST_ARCH variable to that architecture. Were you expecting anything different?

From Google it looks like the msys 32-bit Perl should have archname=i686-msys-thread-multi-64int so I can add that too.

[Andrew Johnson (anj)]

This is my proposed change for the arch-detection:

diff --git a/src/tools/EpicsHostArch.pl b/src/tools/EpicsHostArch.pl
index e8e49bc5e..90a128efa 100644
--- a/src/tools/EpicsHostArch.pl
+++ b/src/tools/EpicsHostArch.pl
@@ -36,6 +36,8 @@ sub HostArch {
         return 'linux-arm' if m/^arm-linux/;
         return 'windows-x64' if m/^MSWin32-x64/;
         return 'win32-x86' if m/^MSWin32-x86/;
+ return 'windows-x64-mingw' if m/^x86_64-msys/;
+ return 'win32-x86-mingw' if m/^i686-msys/;
         return "cygwin-x86_64" if m/^x86_64-cygwin/;
         return "cygwin-x86" if m/^i[3-6]86-cygwin/;
         return 'solaris-sparc' if m/^sun4-solaris/;

It should be relatively straightforward to create a CONFIG.windows-x64-mingw.win32-x86-mingw file, which would let you do cross-builds of the 32-bit target from the 64-bit host-arch. That seems like the best way to handle your particular configuration, take a look at the CONFIG.win32-x86-mingw.win32-x86-mingw-debug file as a model if you're interested.

[Ralph Lange (ralph-lange)]

But these are *not* cross-builds.

With the mingw32 tool chain, you are using a 32bit compiler executable to build 32bit artifacts.
It's the same as on Intel based Linux or Windows, where you can run the 32bit native compilers on a 64bit system.

Also, this is not a particular configuration, but the out-of-the-box standard on MSYS2/MinGW systems. The installer creates two icons "MinGW32 shell" and "MinGW64 shell" that start MSYS2 with the appropriate toolchain setting.

I was assuming the EpicsHostArch script to be a user convenience service, trying to determine the correct EPICS host architecture for the context it is run in. The environment settings in a "MinGW32 shell" or "MinGW64 shell" clearly indicate which toolchain is to be used. IMHO, the EpicsHostArch script should output the setting for the intended toolchain.

[Ralph Lange (ralph-lange)]

Your proposal is actually not producing the correct output.

On a plain MSYS2 system without any of the MinGW toolchains configured (that's the third icon "MSYS2 shell" the installer creates) setting the EPICS Host arch to any of the MinGW variants will not work. Plain MSYS2 is not an EPICS supported platform.

[Ralph Lange (ralph-lange)]

The situation on plain Windows is very similar.
On a 64bit Windows system (the standard) running the vcvarsall.bat tool sets up your shell for using the 32bit or 64bit native Visual Studio compiler.
In this case, again, there is no cross compiling involved. It's either the 32bit compiler producing executables for 32bit or the 64bit compiler producing 64bit executables. (You could also set up cross compiling 32->64 or 64->32, but that's a separate topic.)

Even after you have run vcvarsall.bat setting up everything for 32bit native compiling, EpicsHostArch will report that the architecture is 'windows-x64'. Which is technically correct, but clearly and obviously not the architecture that you want EPICS to use.
(This is the situation for all our CI builds, where we first run vcvarsall.bat and after that still can't use the EpicsHostArch script.)

[Ralph Lange (ralph-lange)]

Since the current behavior is not technically wrong, and your suggestion would introduce an IMHO worse wrong setting, I propose to leave things as they are for this upcoming release, and discuss the intentions and workings of the EpicsHostArch script more broadly.

I believe a user tool should focus on user friendliness and try to come up with the best setting for the EPICS build in the context it is being called from.

[Ben Franksen (bfrk)]

FWTW, I largely agree with Ralph. EpicsHostArch scripts is for
development purposes and should strive to guess what the devloper
intends. It obviously cannot be perfect, given such a "specification",
but that's no reason not to try harder. If taking the user's envoronment
into account is what is needed then so it should.

[Andrew Johnson (anj)]

I agree that "Further discussion" as Debian puts it is probably in order here, and we should decide what we want the EpicsHostArch.pl script to be able to do (and who's going to maintain it).

Note that if you don't set EPICS_HOST_ARCH at all and just run 'make' the EPICS build system (in configure/CONFIG) will call the script to set it for the duration of the build. That works reasonably well for the Unix derivatives, but Windows is a very different beast and I don't believe we could hope to detect and support all possible Windows build environments automatically.

Personally (as you probably know) I use Strawberry Perl (as installed by Chocolatey), which comes with copies of GNUmake and gcc. That plus Visual Studio is all I need to be able to build for win32-x86, windows-x64 and windows-x64-mingw so there's no way that the EpicsHostArch.pl script could tell which of those 3 targets I want it to build. I have a series of .bat files that I run beforehand to choose (I don't use the startup .bat files); Janet used to have multiple desktop icons for starting the command shell with different environments. Actually it looks like I stopped bothering with win32-x86 at all when IT upgraded me to Windows 10, does anyone still use 32-bit CPUs to run Windows?

[Ralph Lange (ralph-lange)]

I would say, in cases where the user runs the compiler setup (vsvarsall or equivalent script) first, EpicsHostArch could look at a few environment variables and figure out the intended setting in most cases.
It would not detect all possible cases on all possible platforms, certainly, but should the inability to reach perfection keep us from improving it?

Since you do run your compiler setup batch beforehand, I am sure EpicsHostArch could perfectly tell which of those targets you want to build in your Strawberry field.

Older free VisualStudio versions (2008, 2010, 2012) that don't offer the 64bit compiler are the main 32bit reasons I can think of. Not sure about legacy/embedded systems - are those running Windows?

[Andrew Johnson (anj)]

Seems reasonable.

BTW, do you need to make any CONFIG_SITE or CONFIG changes to build on Msys2? If so that might suggest that we should add separate {win32-x86,windows-x64}-msys targets, but you haven't mentioned that so I guess we're probably good with what we have.

[Greg Huff (hufffhi)]

so if my architecture is "x86_64-msys-thread-multi" should I set EPICS_HOST_ARCH to windows-x64?

[Ralph Lange (ralph-lange)]

windows-x64-mingw

Complete rewrite of the Windows README is under review.
Find the PullRequest here:
https://epics--9.org.readthedocs.build/projects/how-tos/en/9/getting-started/installation-windows.html