[19.10 FEAT] Provide optimized libatlas libraries for different types of z Systems

[bugproxy]

libatlas is an highly optimized linear algebra library providing
algorithms which are being used as the base for important math
frameworks as e.g. Python NumPy. As such it will play a major role
for e.g. AI applications.

The performance of libatlas heavily depends on being optimized for
certain hardware levels. On IBM Z the introduction of the SIMD
instructions with z13 makes a huge difference for such
calculations. With the following machine (IBM z14) the SIMD
instruction set got substantially enhanced to support more data types
(single precision float) and can be expected to be subject of further
changes in the future.

In order to benefit from the instructions set improvements in libatlas
it is required to:

1. Compile libatlas with a GCC version being able to exploit the
   instruction set enhancements and use the proper -march/-mtune
   options for compilation.

2. Run the libatlas builtin tuning framework to determine which
   implementation of certain algorithms works best for that particular
   hardware level.

Doing this e.g. on IBM z14 with a libatlas version enabled for running
proper tuning on IBM Z provides a speedup of up to 5x compare to the
version currently provided with Ubuntu 18.04.

However, since the base level for Ubuntu currently is zEC12 no tuned
libatlas version for z13 or higher is part of the distro.

This request is about building libatlas with IBM provided tuning
results for z13 and z14 and install these in parallel to the default
zEC12 version. The well-established ELF capability depending load
mechanism in ld.so can be used to make sure that the proper version
will be picked at load time of the library. So it is basically a
matter of adjusting the build process of libatlas to enable creating
several binary variants of the library and place them in the proper
paths during installation.

Addl. information and tar balls will follow to provide a solution for this situation

[Dimitri John Ledkov (xnox)]

I'm expecting for you to provide tarballs similar to:

$ ls debian/archdefs/s390x/*.bz2
debian/archdefs/s390x/IBMz964.tar.bz2
debian/archdefs/s390x/IBMz1264.tar.bz2

but for z13, and z14. Also please check the IBMz1264.tar.bz2 tarball if you can or want to change the performance there.

If there are any additional patches required to cherrypick from upstream, please point them out.

And any desired compiler flags, and configuration options.

For the tuned builds, we can use things like -march=z13 -O3 and so on, as mentioned in your bug report.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-20 03:52 EDT-------
*** Bug 173086 has been marked as a duplicate of this bug. ***

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2019-02-26 11:00 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2019-02-26 11:01 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2019-02-26 11:02 EDT-------

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-02-26 11:22 EDT-------
> I'm expecting for you to provide tarballs similar to:
>
> $ ls debian/archdefs/s390x/*.bz2
> debian/archdefs/s390x/IBMz964.tar.bz2
> debian/archdefs/s390x/IBMz1264.tar.bz2
>
> but for z13, and z14. Also please check the IBMz1264.tar.bz2 tarball if you
> can or want to change the performance there.

Done. See the attachments above for the z13 and z14 tarballs. For the existing IBMz1264.tar.bz2, I suggest to leave it as is for now.

> If there are any additional patches required to cherrypick from upstream,
> please point them out.

Yes, there are patches, but not all of them are upstream. I attached them above instead. Those apply to a vanilla source tree extracted from atlas3.10.3.tar.bz2. (Upstream ATLAS doesn't currently work on IBM Z. This is still under debug.)

> And any desired compiler flags, and configuration options.

Just use the flags as defined by the patches, i.e., "-march=z13/z14 -mtune=z13/z14 -O2".

[Dimitri John Ledkov (xnox)]

The patches look good, and apply almost cleanly (had to rebase the z14 conf one, due to distro-patch to add generic type, but that's fine).

The tuning tarballs look good.

Note to sell, this is a request to compile:
MACHTYPE IBMz12 ISA 1
MACHTYPE IBMz13 ISA VXZ
MACHTYPE IBMz14 ISA VXZ2

Note to self that ISA values are changed with the z14 patch:

$ gcc foo.c; ./a.out
FPV3D16MAC: 16384
VXZ: 4

$ quilt push
Applying patch debian/patches/0006-Add-IBM-z14-support.patch
patching file CONFIG/include/atlconf.h
patching file CONFIG/src/Makefile
patching file CONFIG/src/atlcomp.txt
patching file CONFIG/src/backend/Make.ext
patching file CONFIG/src/backend/archinfo_linux.c
patching file CONFIG/src/backend/probe_vxz2.c
patching file CONFIG/src/probe_comp.c
patching file include/atlas_prefetch.h
patching file include/atlas_simd.h

Now at patch debian/patches/0006-Add-IBM-z14-support.patch

$ gcc foo.c; ./a.out
FPV3D16MAC: 32768
VXZ: 8
VXZ2: 4

Thus need to make sure to update -V flags for arm, and use the patched valued for s390x.

[Dimitri John Ledkov (xnox)]

Ditto SSE1+SSE2 ISA values changed:

$ ./a.out
SSE1+SSE2: 768
FPV3D16MAC: 16384
VXZ: 4

$ ./a.out
SSE1+SSE2: 1536
FPV3D16MAC: 32768
VXZ: 8
VXZ2: 4

[Dimitri John Ledkov (xnox)]

I can ship the patches (I had to rebase them a bit, please double check them) and the archdefs tarballs.

I don't think I will be able to build z14 optimized libraries in launchpad and ship in the ubuntu archive. atlas cannot be cross-compiled, and the build at the moment executes just generated binaries..... which segfaults with illegal instructions, as launchpad is a z13 mainframe with z14 instructions not available. And we require source-only uploads, and all binary builds in launchpad as shipped into ubuntu archive. It is unlikely that we will change mainframe at canonical anytime soon. At best I can just leave the conditional codepaths in packaging, such that self-rebuilding the package is easier and defaults to attempting to build z14 optimized libraries. Even if it was possible to "cross-compile" for z14, I would be concerned about shipping untested builds of z14 code, as we wouldn't be able to exercise the testsuite during build.

I should be able to build z13 optimized libraries, however some of them are failing to build from source - have you tested enabling -march=z13, the optimized instructions, with the provided archdefs on an Ubuntu system? do you have a full build log?

the s390x build log is long, so here are some search markers:
Search for: make -C build-z13
that's where the z13 optimized compilation starts
Then search for: cd KERNEL ; make -f dMakefile dlib

Thing go really weird there:
make[9]: Entering directory '/<<PKGBUILDDIR>>/build-z13/src/blas/gemm/KERNEL'
/usr/bin/gcc -DL2SIZE=33554432 -I/<<PKGBUILDDIR>>/build-z13/include -I/<<PKGBUILDDIR>>/build-z13/..//include -I/<<PKGBUILDDIR>>/build-z13/..//include/contrib -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DATL_OS_Linux -DATL_ARCH_IBMz13 -DATL_CPUMHZ=3446 -DATL_VXZ -DATL_USE64BITS -DATL_GAS_S390 -m64 -DATL_DYLIBS -DWALL -DATL_UCLEANM -DATL_UCLEANN -DATL_UCLEANK -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -march=z13 -m64 -fPIC -c ATL_dNBmm_b0.c
In file included from ATL_dNBmm_b0.c:59:
ATL_dNBmm_b0.c: In function ‘ATL_dJIK72x72x72TN72x72x0_a1_b0’:
/<<PKGBUILDDIR>>/build-z13/..//include/atlas_simd.h:438:29: error: expected ‘;’ before ‘double’
    #define ATL_VTYPE vector double
...

Is there something broken with the compiler? Did i miss-apply / miss-rebase the provided patches?

The net result is that liblapack_atlas.a and libf77blas.a are not compiled, and then at most I can ship optimized libatlas.so and libcblas.so only, but not other libraries. And all other architectures still build fine, which is good.

You can download the source package for local experimentation using:

$ dget http://ppa.launchpad.net/xnox/nonvirt/ubuntu/pool/main/a/atlas/atlas_3.10.3-8ubuntu1+ppa3.dsc
$ dpkg-source -x atlas*.dsc
$ cd atlas-3*
$ sudo apt install build-essential
$ sudo apt build-dep ./
$ dpkg-buildpackage -b .

If you do above on a z14 machine, it should attempt to build all three variants of the library: for z12, z13, and z14.

URL to build record: https://launchpad.net/~xnox/+archive/ubuntu/nonvirt/+build/16494235
Build log is linked from there: https://launchpadlibrarian.net/415026393/buildlog_ubuntu-disco-s390x.atlas_3....

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-14 12:34 EDT-------
The should be an -mzvector option somewhere to enable the high level vector data types. Wasn't that part of the libatlas flags for IBM Z?

Would it suffice if we would run the libatlas z14 testsuite and provide you with the results?

I assumed that during a libatlas build with pre-tuned files no library code needs to be executed during build. We will have a look what's going on there.

[Dimitri John Ledkov (xnox)]

re:z14 - That would not sufice, as it means that it would not be part of regular archive rebuilds, autopkgtest testing of itself, reverse dependencies, and like toolchain/kernel. I am sure we can ship it, but the stability and quality concerns will be degraded a lot. And it means it will be out of scope for canonical to maintain / fix. I'm sure we will drop z14 builds if they start failing to build or are reported as broken, given lack of our z14 access on the launchpad internal infrastructure.

about flags. I'll check if -mzvector can be added, and if that helps. Can that go for all three builds? z12, z13, z14? Note that during configure stage (as seen in debian/rules) we do override cflags, flags with distro defaults.

For z12 build we use:
cd build && ../configure --prefix="/home/ubuntu/atlas/debian/tmp" --incdir="/home/ubuntu/atlas/debian/tmp/usr/include/s390x-linux-gnu/" --libdir="/home/ubuntu/atlas/debian/tmp/usr/lib/s390x-linux-gnu/" --shared -D c -DWALL -Ss f77lib "-L/usr/lib/gcc/s390x-linux-gnu/8/ -lgfortran -lgcc_s -lpthread" -Ss pmake 'make' -v 2 -b 64 -A IBMz12 -V 1 -t 0 --cc="cc" --cflags="-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/home/ubuntu/atlas=. -fstack-protector-strong" -C acg gcc -F acg "-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/home/ubuntu/atlas=. -fstack-protector-strong" -C if gfortran -F if "-g -O2 -fdebug-prefix-map=/home/ubuntu/atlas=. -fstack-protector-strong" -Ss ADdir ../../debian/archdefs/s390x --cripple-atlas-performance

z13 adds -A IBMz13 -V 8 config options, and -march=z13 to all the *flags
z14 adds -A IBMz14 -V 4 config options, and -march=z14 to all the *flags
and no -mzvector is not used at the moment.

When I attemped the build, the build stage does compile temporary binaries, and does try to execute them.

Let me try to get z13 build going, and i'll provide you the logs about the failing to build z14-on-z13.

[Dimitri John Ledkov (xnox)]

I see in gcc docs that -mzvector is z13+

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-15 04:29 EDT-------
(In reply to comment #18)
...
> z13 adds -A IBMz13 -V 8 config options, and -march=z13 to all the *flags
> z14 adds -A IBMz14 -V 4 config options, and -march=z14 to all the *flags
> and no -mzvector is not used at the moment.

These two should -mzvector as well I think. But GCC will not complain about -mzvector being used with -march=zEC12. -mzvector does only enable certain language extension and will not affect code generation.

> When I attemped the build, the build stage does compile temporary binaries,
> and does try to execute them.

Do these temporary binaries need to use the same build options as the library? If these are just helper binaries for the build perhaps there is a separate CFLAGS variable just for those?

[Dimitri John Ledkov (xnox)]

Added -mzvector, fixed up a few more packaging bits, and it looks like z13 optimized libraries are fully built.

Please test:

sudo add-apt-repository ppa:ci-train-ppa-service/3679
sudo apt-get update

On Ubuntu Disco (19.04), and let me know if you see performance improvements on a z13+. It is still built in a lxd container, in a KVM VM, on z13 mainframe, in Launchpad. And I do still see that build logs are running a few binaries to find values for things (which looks odd). But hopefully, the performance is better with this build now.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-18 14:14 EDT-------
> Added -mzvector, fixed up a few more packaging bits, and it looks like z13
> optimized libraries are fully built.
Hm, I've tried /usr/lib/s390x-linux-gnu/z13/libatlas.so.3.10.3. The sgemm/dgemm kernels from this library don't use any vector instructions, thus the performance for dgemm is off by a factor of 2. Something must have gone wrong in the build, although I couldn't determine the problem from the build log for "3679 - 2019-03-15".

(BTW, I added a symlink "z14" -> "z13" in /usr/lib/s390x-linux-gnu in order to test the library on a z14. Not sure if that's the intended use or not, but it certainly worked.)

[Dimitri John Ledkov (xnox)]

I would expect for z13 libraries to be used automatically on z14, when /z14 ones are missing.... certainly that's the expectation to use the "best possible" libraries without any extra symlinks.

Did you see that z13 libraries were _not_ used? I'll try to test that on a z14 machine.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-19 04:30 EDT-------
I think the subdirs for z13 should be vx and for z14 vx/vxe. These are the hardware facilities as stated in /proc/cpuinfo plus the commitment from the Linux kernel to save and restore vector registers. For z14 both facilities need to be present. Glibc builds search paths including all of the current facility strings as subdirs. The longest path holding a library wins.

I'm not sure how the policy is regarding the arch strings. While z13 and z14 are supported as subdirs of the search path I would not expect the search to go downwards automatically. With IBM Z it would be ok to do this but for other platforms it might not be possible to give all the CPU levels a strict order where more recent machines automatically support all features of previous generations.

To be honest I've never looked into how the library search paths work together with the Debian/Ubuntu multiarch path setup. But I would expect that to work well together - right?!

However, currently vx and vxe or not part of the "important" hwcaps. So Glibc will not use these when searching for libraries. We might need a small patch for Glibc. Stefan, could you please have a look.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-20 05:37 EDT-------
As already mentioned in Andreas' comment, vx and vxe is currently not part of the "important" hwcaps.
I've just posted the patch "S390: Mark vx and vxe as important hwcap." (https://www.sourceware.org/ml/libc-alpha/2019-03/msg00401.html) for upstream glibc.
Once it is committed I will backport it to glibc release 2.27 (Ubuntu 18.04) .. 2.29 (Ubuntu 19.04) branches.
Will those commits on glibc-release-branches automatically picked up in order to be available in your libc6 packages?
Or do we need to open extra bugzillas?

The correct locations for the different tuned libraries are:
-for z14 with vxe-support: /usr/lib/s390x-linux-gnu/vxe/
-for z13 with vx-support: /usr/lib/s390x-linux-gnu/vx/
-for all without vector-support: /usr/lib/s390x-linux-gnu/

You can also test this without the glibc-patch by setting LD_HWCAP_MASK:
export LD_HWCAP_MASK=0x2872
Note: 0x2800 belongs to vx and vxe hwcaps and 0x72 belongs to the previous setting of HWCAP_IMPORTANT (zarch, ldisp, eimm, dfp).
While LD_HWCAP_MASK is set, run ldconfig and afterwards start your program which is then choosing the appropriate library.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-21 04:47 EDT-------
I've just committed the glibc patch "S390: Mark vx and vxe as important hwcap." (https://sourceware.org/git/?p=glibc.git;a=commit;h=61f5e9470fb397a4c334938ac5a667427d9047df) to glibc-master-branch
and cherry-picked it on the glibc 2.27, 2.28, 2.29 release branches.

Please let us know if you automatically pick those commits or if we have to open an extra request.

[Dimitri John Ledkov (xnox)]

@STLI please open and mirror a new request for glibc vx vxe patch. With the links to git commits on relevant branches.

However, it is trivial for me to ship symlinks and/or to change the directories where the optimized binaries are shipped. To both z13 and z14, or to both z13 & vx name, or just vx name in the mean time. As whatever is needed by current glibc in Ubuntu to start using them. So landing glibc patch is not a blocking prerequisite here.

[Dimitri John Ledkov (xnox)]

About performance.

So this builds are still not good? based on the earlier comment:
"""
Hm, I've tried /usr/lib/s390x-linux-gnu/z13/libatlas.so.3.10.3. The sgemm/dgemm kernels from this library don't use any vector instructions, thus the performance for dgemm is off by a factor of 2. Something must have gone wrong in the build, although I couldn't determine the problem from the build log for "3679 - 2019-03-15".
"""

Can you perchance provide a full build-log, of how you build the library locally for z13 (without any debian/ubuntu packaging) and get the exepcted performance? Such that i can dig into that build log to figure out what i am doing wrong?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-22 15:02 EDT-------
> Can you perchance provide a full build-log, of how you build the library
> locally for z13 (without any debian/ubuntu packaging) and get the exepcted
> performance? Such that i can dig into that build log to figure out what i am
> doing wrong?
After digging a bit more, I don't think that will be necessary. I experimented with this package:
http://ppa.launchpad.net/xnox/nonvirt/ubuntu/pool/main/a/atlas/atlas_3.10.3-8ubuntu1+ppa3.dsc

First I had a bit of trouble with the build, because the "rules" file doesn't specify the -mzvector flag in the compile options. (Is this the latest source you've been using?) I patched the rules file accordingly, adding variables with additional config options for z13 and z14, where the z13 version looks like this:

CONF_Z13 := -A IBMz13 -V 8 \
-F acg "$(CPPFLAGS) $(CFLAGS) -march=z13 -mzvector" \
-F xc "$(CPPFLAGS) $(CFLAGS)" \
-F if "$(FFLAGS) -march=z13"

And using the variables like this:

ifeq ($(Z13),yes)
mkdir -p build-z13
cd build-z13 && ../configure $(CONFIGURE_FLAGS) $(CONF_Z13)
endif

Note that I ask Atlas to exclude the "march" and "mzvector" options for the "xc" group of compilations. This might be a first step towards "cross-compilation".

After that, the build worked on z14 and created vector-enabled binaries for z13 and z14 which seem to yield reasonable performance. But then the "*.install" files gave me trouble, and instead of completing the build of the .deb files, I got error messages like this:

make[1]: Entering directory '/home/arnez/atlas-3.10.3'
dh_install
dh_install: Cannot find (any matches for) "yes" (tried in ., debian/tmp)

dh_install: Cannot find (any matches for) "yes" (tried in ., debian/tmp)
...

This obviously originates from the "${Z13}" and "${Z14}" in the "*.install" files, e.g.:

[s390x] ${Z13} build/libatlas.so.* /usr/lib/${DEB_HOST_MULTIARCH}/z13

After removing all occurrences of "${Z13}" and "${Z14}", the build completed with a warning "dh_missing: usr/lib/s390x-linux-gnu/libsatlas.so exists in debian/tmp but is not installed to anywhere."

However, the resulting ".deb" files contained the wrong shared libraries for z13 and z14! Instead of using the libraries built under "build-z13" and "build-z14", the library under "build" was replicated three times. This is obviously due to an error in the "*.install" files as well, where the build-z13 and build-z14 aren't referenced at all.

I hope this helps.

[Dimitri John Ledkov (xnox)]

On Fri, 22 Mar 2019 at 19:15, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2019-03-22 15:02 EDT-------
> > Can you perchance provide a full build-log, of how you build the library
> > locally for z13 (without any debian/ubuntu packaging) and get the exepcted
> > performance? Such that i can dig into that build log to figure out what i am
> > doing wrong?
> After digging a bit more, I don't think that will be necessary. I experimented with this package:
> http://ppa.launchpad.net/xnox/nonvirt/ubuntu/pool/main/a/atlas/atlas_3.10.3-8ubuntu1+ppa3.dsc
>

... which is an obsolete source that never even managed to compile,
and not install any libraries ever.
And it has been deleted and longer published, but not yet garbage
collected by launchpad.

And below, you seem to be repeating the steps that were already done.
Please don't use that source package....

The source and binaries, that we are currently evaluating are from this ppa:
"""
Please test:
sudo add-apt-repository ppa:ci-train-ppa-service/3679
sudo apt-get update
"""
You can enable source repository for that ppa too, to "apt-get source atlas".

Or if you need more URLs to build logs please see:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3679
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3679/+build/16500359
http://ppa.launchpad.net/ci-train-ppa-service/3679/ubuntu/
http://ppa.launchpad.net/ci-train-ppa-service/3679/ubuntu/pool/main/a/atlas/

However, you prefer to fetch / analyse that. (launchpad PPA, build
record, apt-get archive, the pool location of atlas)

> First I had a bit of trouble with the build, because the "rules" file
> doesn't specify the -mzvector flag in the compile options. (Is this the
> latest source you've been using?) I patched the rules file accordingly,

It is not =) note the compilation and installations were fixed up for
the better in the above ci-train-ppa-service ppa....

> adding variables with additional config options for z13 and z14, where
> the z13 version looks like this:
>
> CONF_Z13 := -A IBMz13 -V 8 \
> -F acg "$(CPPFLAGS) $(CFLAGS) -march=z13 -mzvector" \
> -F xc "$(CPPFLAGS) $(CFLAGS)" \
> -F if "$(FFLAGS) -march=z13"
>
> And using the variables like this:
>
> ifeq ($(Z13),yes)
> mkdir -p build-z13
> cd build-z13 && ../configure $(CONFIGURE_FLAGS) $(CONF_Z13)
> endif
>
> Note that I ask Atlas to exclude the "march" and "mzvector" options for
> the "xc" group of compilations. This might be a first step towards
> "cross-compilation".
>

I'll try above. However, note that the above PPA, there is -march=z13
-mzvector flags passed, yet I was told earlier that it did not produce
vector enabled binaries.

I'll see if "xc" group of flags helps with building z14 atlas on z13,
this is helpful. My questions are still not answered as both
'-march=z13 and -mzvector' flags are in used in that ppa, and it is
claimed that no vector optimized libraries are produced. Where the
wrong binaries tested, i.e. those from the generic Ubuntu archive,
rather than from the ci-train-ppa?

1) Please re-review sources from the correct location
(ci-tr...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-03-25 10:40 EDT-------
> 1) Please re-review sources from the correct location
> (ci-train-ppa-service urls above)

OK, these look better, but the main issue remains the same:

$ cd atlas-3.10.3/debian
$ grep build-z13 *.install
$ grep build-z14 *.install
$

It seems that none of the files in "build-z13" or "build-z14" are installed by the install rules. Everything is taken from the "build" directory instead.

> 2) Please re-review binaries from the correct location to see if they
> are vector enabled or not.

I've been using the binaries from the PPA all along. This is what I (still) see:

$ cd /usr/lib/s390x-linux-gnu/
$ dpkg -S z13/libatlas.so.3.10.3
libatlas3-base:s390x: /usr/lib/s390x-linux-gnu/z13/libatlas.so.3.10.3
$ diff -s libatlas.so.3.10.3 z13/libatlas.so.3.10.3
Files libatlas.so.3.10.3 and z13/libatlas.so.3.10.3 are identical
$ objdump -d z13/libatlas.so.3.10.3 | grep vfma | wc
0 0 0
$

> 3) Whilst doing above, please paste the output of $ apt-cache policy
> libatlas3-base

$ apt-cache policy libatlas3-base
libatlas3-base:
Installed: 3.10.3-8ubuntu1
v Candidate: 3.10.3-8ubuntu1
Version table:
*** 3.10.3-8ubuntu1 500
500 http://ppa.launchpad.net/ci-train-ppa-service/3679/ubuntu disco/main s390x Packages
100 /var/lib/dpkg/status
3.10.3-8 500
500 http://ports.ubuntu.com disco/universe s390x Packages
$

> ps. i am sorry for causing any confusion due to change of the location
> of where the builds are located.

Yeah, indeed, I was a bit confused about the location of the source package. Thanks for clearing that up.

Regarding z14 support, I'll experiment with "cross-compiling" (z13 -> z14) and see how far I can get with that.

[Dimitri John Ledkov (xnox)]

*sigh* well, indeed the previous packaging does a lot of work, and doesn't install anything in the end. Re-uploaded atlas with below attached patch, which should hopefully install optimized z13 builds in the right place now.

[Dimitri John Ledkov (xnox)]

3.10.3-8ubuntu2 is now published in the same repository, and should be better. https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3679/+build/16548143

at least the standard and z13 libraries are now of different size, meaning are different. Hopefully they are now vector optimized for z13.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-04-01 11:35 EDT-------
> 3.10.3-8ubuntu2 is now published in the same repository, and should be
> better.
Right. I tested libatlas3-base and can confirm that it contains the vector-enabled library in the z13 directory. Now the library performs as expected.
I'm still trying to find a way of "cross-compiling" ATLAS in order to create z14 binaries as well. This is particularly relevant for single-precision FP, where the performance can be improved by a significant factor from z13 to z14.

[Dimitri John Ledkov (xnox)]

Updated packaging to install into /vx & /vxe paths.
Only z13 optimized libraries will be produced by launchpad, however rebuilding the package locally on a z14 mainframe should generate both z13 and z14 builds.

I consider this done. For z14 optimized libraries please open a new ticket when launchpad gains z14 hardware, or when "cross-compiling" for z14 is possible on an z13 system.

[Launchpad Janitor (janitor)]

This bug was fixed in the package atlas - 3.10.3-8ubuntu3

---------------
atlas (3.10.3-8ubuntu3) eoan; urgency=medium

  * Imprort cherrypicked upstream patches to correct z13, z14 SIMD vector
    instructions.
  * Add tarballs for z13 and z14 tuning.
  * Attempt to build for z13 and z14, install into vx and vxe directories.
  * Whilst neither z14 hw is availabl, nor cross-compilation, only vx
    optimized binaries are provided on s390x. LP: #1814796

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Mar 2019 17:48:48 +0000

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-05-06 05:41 EDT-------
IBM bugzilla status -> closed, Fix Released for disco in support of z13.
For z14 support a new ticket will be opened against Ubuntu 19.10.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2019-06-26 11:19 EDT-------
> This bug was fixed in the package atlas - 3.10.3-8ubuntu3
Sorry, this hasn't worked.

While the above package is indeed contained in the pool (at http://ports.ubuntu.com/ubuntu-ports/pool/universe/a/atlas/), it doesn't appear in any package list I checked (disco, disco-updates, disco-backports, disco-security, disco-proposed).

This should be fixed. At this time we probably want to place it in disco-updates, right?

But the package itself has a few issues as well. It contains the spurious directories /usr/lib/s390x-linux-gnuvx and /usr/lib/s390x-linux-gnu/atlasvx whose contents aren't used anywhere. And it lacks alternatives definitions for /usr/lib/s390x-linux-gnu/vx/libblas.so.3 and friends. The associated development package libatlas-base-dev does contain the missing "vx" directory in /usr/lib/s390x-linux-gnu, but only with dead links in it.

I believe the packages should be changed such that for each of the libraries libblas.so.3, libcblas.so.3, and liblapack.so.3:
- the vx-specific version is installed into /usr/lib/s390x-linux-gnu/vx/atlas
- the non-vx version is installed into /usr/lib/s390x-linux-gnu/atlas
- an alternative is established from /usr/lib/s390x-linux-gnu/<lib> to /usr/lib/s390x-linux-gnu/atlas/<lib>
- an alternative (slave) is established from /usr/lib/s390x-linux-gnu/vx/<lib> to /usr/lib/s390x-linux-gnu/vx/atlas/<lib>

Also, appropriate versions of libatlas.so.3 and liblapack_atlas.so.3 should be directly installed into /usr/lib/s390x-linux-gnu/vx and /usr/lib/s390x-linux-gnu. Maybe the same applies to libf77blas.so.3, or maybe this should also be handled with alternatives.