[19.10 FEAT] Provide optimized libatlas libraries for different types of z Systems ( z13+z14).

[bugproxy]

Provide optimized libatlas libraries for different types of z Systems ( z13+z14).

Follow on Feature based on
z13
bugzilla -> https://bugzilla.linux.ibm.com/show_bug.cgi?id=173368
LP -> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/1814796

z14
bugzilla -> https://bugzilla.linux.ibm.com/show_bug.cgi?id=177415
LP -> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/1827865

More information will follow.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-07-24 08:37 EDT-------
For previous work on this topic, see:
<https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/1814796>
The resulting package had some issues. I suggest the following changes:

1. Drop the spurious directories /usr/lib/s390x-linux-gnuvx and
/usr/lib/s390x-linux-gnu/atlasvx.
2. For each of the libraries libblas.so.3, libcblas.so.3, and
liblapack.so.3:
- Install the "vx"-specific library version into
/usr/lib/s390x-linux-gnu/vx/atlas.
- Install the non-"vx" version into /usr/lib/s390x-linux-gnu/atlas.
- Establish an alternative from /usr/lib/s390x-linux-gnu/<lib> to
/usr/lib/s390x-linux-gnu/atlas/<lib>.
- Establish an alternative (slave) from
/usr/lib/s390x-linux-gnu/vx/<lib> to
/usr/lib/s390x-linux-gnu/vx/atlas/<lib>.
3. Install appropriate versions of libatlas.so.3 and
liblapack_atlas.so.3 directly into /usr/lib/s390x-linux-gnu/vx and
/usr/lib/s390x-linux-gnu. Maybe the same applies to libf77blas.so.3,
or maybe this should also be handled with alternatives.

And in addition, this new feature request is meant to address IBM z14
support as well. For this, additional changes are needed:

4. Add "cross-compilation" support for ATLAS. I'll attach patches for
that.
5. Add "vxe" variants for each of the "vx" variants above and place the
z14 binaries there.

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2019-07-24 09:08 EDT-------

Add support for building ATLAS without running any target code. This requires some additional files in archdefs that would otherwise be built during various tuning steps. Thus I'll also attach new archdefs for IBM z13 and IBM z14.

Note that the patch doesn't automatically enable cross compilation. To activate it and disable tuning at
build time, add the option "-Si archdef 2" when running "configure".

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2019-07-24 09:11 EDT-------

This version of the IBM z13 archdefs contains additional files for cross-compilation.

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2019-07-24 09:13 EDT-------

Cross-compile-ready archdefs for IBM z14.

[Dimitri John Ledkov (xnox)]

I am attempting to enable all of this work.

However, I'm getting a build-failure when building z14 code on z13, still. Maybe my configure flags are bad, and/or more tuned headers are missing.

I wonder if for the z14 build, I should be setting -march=z13 -mtune=z14 in XCCFLAGS such that xemit_mm (and the like) binaries can be executable. Please see:

https://launchpadlibrarian.net/437023671/buildlog_ubuntu-eoan-s390x.atlas_3.10.3-8ubuntu4_BUILDING.txt.gz

make[5]: *** [Makefile:390: dinstall] Illegal instruction (core dumped)
make[5]: *** [Makefile:684: dmvninstall] Illegal instruction (core dumped)

/<<PKGBUILDDIR>>/build-z14/..//include/atlas_kern3.h:34:1: fatal error: atlas_dNCmm.h: No such file or directory
/<<PKGBUILDDIR>>/build-z14/..//include/atlas_kern3.h:34:1: fatal error: atlas_dNCmm.h: No such file or directory
/<<PKGBUILDDIR>>/build-z14/..//include/atlas_kern3.h:34:1: fatal error: atlas_dNCmm.h: No such file or directory
/<<PKGBUILDDIR>>/build-z14/..//include/atlas_kern3.h:34:1: fatal error: atlas_dNCmm.h: No such file or directory

I'll upgrade the build with new tuning tarballs and correct paths for the z13 build.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-08-12 10:17 EDT-------
> [...]
> I wonder if for the z14 build, I should be setting -march=z13 -mtune=z14 in
> XCCFLAGS such that xemit_mm (and the like) binaries can be executable.
Yes, sorry, I didn't mention that before. Cross compile support relies on XCC to be compiling for the build machine, not the target machine. Thus I suggest to omit the "-march" and "-mtune" options in XCCFLAGS.

[Dimitri John Ledkov (xnox)]

So our current call to configure is this:

cd build-z14 && ../configure --prefix="/<<PKGBUILDDIR>>/debian/tmp" --incdir="/<<PKGBUILDDIR>>/debian/tmp/usr/include/s390x-linux-gnu/" --libdir="/<<PKGBUILDDIR>>/debian/tmp/usr/lib/s390x-linux-gnu/" --shared -D c -DWALL -Ss f77lib "-L/usr/lib/gcc/s390x-linux-gnu/9/ -lgfortran -lgcc_s -lpthread" -Ss pmake 'make -j 4' -v 2 -b 64 -A IBMz12 -V 1 -t 0 --cc="cc" --cflags="-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" -C acg gcc -F acg "-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" -C if gfortran -F if "-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" -Ss ADdir ../../debian/archdefs/s390x --cripple-atlas-performance -Si archdef 2 -A IBMz14 -V 4 --cflags="-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -march=z14 -mzvector" -C acg gcc -F acg "-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -march=z14 -mzvector" -C if gfortran -F if "-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -march=z14 -mzvector"

I'm not sure about the right -F flags here. Cause I would assume that for target code we do want -march=z14 -mzvector" but for xccflags we do not. And somehow, I'm not that well versed in atlas configure to figure this out. Also it is suboptimal that we pass and overrride -A cause it makes it confusing to read =)

Let me try to push a build out without -march=z14 as a step in the right direction.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-08-15 12:13 EDT-------
> So our current call to configure is this:
> [...]
These configure options look a bit odd. They contain -A IBMz12 as well as -A IBMz14, similarly with -V, -C {acg,if}, -F {acg,if}, and --cflags. I believe the last occurrence of each option overrides all previous ones, but I'd prefer not to exploit that. Better split out common options and add the specific ones just for the appropriate CPU level.
More importantly, the host compile flags (-F xc) are missing. They should be added in order to use the cross-compile logic.
The --cflags are mainly used for bootstrapping the build logic, so no target-specific compiler flags should be included there.

[Dimitri John Ledkov (xnox)]

yeah, currently things are structured for one build per arch.

I guess we should extend this work, and improve packaging to like enable power8/9 builds, and neon, avx2/avx512 builds, etc.

I've now used:

 cd build-z14 && ../configure $(CONFIGURE_FLAGS) -Si archdef 2 -A IBMz14 -V 4 --cflags="$(CPPFLAGS) $(CFLAGS) -march=z14 -mzvector" -C acg gcc -F acg "$(CPPFLAGS) $(CFLAGS) -march=z14 -mzvector" -C if gfortran -F if "$(FFLAGS) -march=z14 -mzvector" -Fa xc '-march=z13'

And the `-Fa xc '-march=z13'` seems to make everything work, despite long and duplicated flags everywhere.

Once that lands, hopefully we will have both z13 and z14 optimized libs out of the box, finally.

[Launchpad Janitor (janitor)]

This bug was fixed in the package atlas - 3.10.3-8ubuntu5

---------------
atlas (3.10.3-8ubuntu5) eoan; urgency=medium

  * Enable z14 build at -march=z14, with cross-compile engine at
    -march=z13. LP: #1837577

 -- Dimitri John Ledkov <email address hidden> Thu, 15 Aug 2019 15:34:03 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-08-20 08:32 EDT-------
IBM bugzilla status -> closed, Fix Released for Eoan