Libjna-jni uses wrong path on package installation on s390x

[Frederik Hartmann]

libjna-jni uses the wrong path for linking the library /usr/lib/s390x-linux-gnu/libjnidispatch.so
on the z Systems s390x architecture, therefore applications relying on this library are unable to link to it, even if the package is installed.
Creating the link manually fixes this problem.

It should be linked from /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so

/sbin/ldconfig.real: Can't link /usr/lib/s390x-linux-gnu//build/libjna-java-D1S9TX/libjna-java-4.2.2/build/native-linux-s390x/libjnidispatch.system.so to libjnidispatch.so

dpkg -S /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so
libjna-jni: /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-07-16 04:28 EDT-------
@Canonical, can someone give an update on that s390 specifix ticket. Many thanks in advance...

[Tiago Stürmer Daitx (tdaitx)]

Could you please add:
1. the specific steps to reproduce this bug
2. which Ubuntu versions that are affected

for now I'm going to set this as incomplete.

[Frederik Hartmann (hartmafk)]

Sure!

1. the specific steps to reproduce this bug

Install package

2. which Ubuntu versions that are affected

I was able to reproduce the bug on the following versions:

16.04 on s390
18.04 on s390

Thanks!

[Frank Heimes (fheimes)]

I ran a little JNI sample program on 16.04 and 18.04 and it worked for me, so obviously the path is fine on my systems(s) - see attachment for details.
Hence we need more details from you - especially the exact steps/commands that let's us reproduce the error situation.

[Frederik Hartmann (hartmafk)]

I have added my current apt upgrade log here to give you a broader context.
The status of my system is added as well.

The issue is here:

/sbin/ldconfig.real: Can't link /usr/lib/s390x-linux-gnu//build/libjna-java-pvBsvy/libjna-java-4.5.1/build/native-linux-s390x/libjnidispatch.system.so to libjnidispatch.so

It SHOULD be the following:

ln -s /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so /usr/lib/s390x-linux-gnu/libjnidispatch.so

The link is not created correctly resulting that the lib libjnidispatch.so is not in the path.
I have tried reinstalling the packages libjna-java libjna-jni multiple times without getting the valid link.

[Tiago Stürmer Daitx (tdaitx)]

By default the package install libjnidispatch under /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so and as far as I can see there's no reason to have a file/link in /usr/lib/s390x-linux-gnu/libjnidispatch.so.

Still, I was able to reproduce the error which indicates that you might have a dangling file somewhere. To do that I simply copied the package's libjnidispatch.system.so to /usr/local/lib and ran ldconfig:

$ sudo cp /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so \
          /usr/local/lib/
$ sudo ldconfig
/sbin/ldconfig.real: Can't link /usr/local/lib//build/libjna-java-Ln6YkD/libjna-java-4.5.1/build/native-linux-x86-64/libjnidispatch.system.so to libjnidispatch.system.so

Running with verbose shows more information:

$ sudo ldconfig --verbose
[other info]
/usr/local/lib:
 /build/libjna-java-Ln6YkD/libjna-java-4.5.1/build/native-linux-x86-64/libjnidispatch.system.so -> libjnidispatch.system.so/sbin/ldconfig.real: Can't link /usr/local/lib//build/libjna-java-Ln6YkD/libjna-java-4.5.1/build/native-linux-x86-64/libjnidispatch.system.so to libjnidispatch.system.so
 (SKIPPED)
[other info]

This should at least show you where to look for the dangling libjnidispatch.system.so (eg. /usr/local/lib in my case).

[Tiago Stürmer Daitx (tdaitx)]

Before removing the "dangling" file you might want to check if it is by any change associated with a package by calling "dpkg -S <path to dangling libjnidispatch>".

[Frederik Hartmann (hartmafk)]

The only "duplicate" link there was, was the link I mentioned before.
If the link is removed the error message is removed, however the library becomes unusable to java applications.

Failed to list up hs_err_pid files
java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-s390x/libjnidispatch.so) not found in resource path

This MIGHT be a potential result of https://bugs.launchpad.net/ubuntu/+source/libjna-java/+bug/1065253 , however, a default jenkins installation will not find libjnidispatch.

This is an issue that I was able to reproduce on at least 3 systems installed from the original sources.

[Tiago Stürmer Daitx (tdaitx)]

Your last comment indicated 2 different errors:

1) The link to libjnidispatch.so
Is the aforementioned link to libjnidispatch.so created by a supported Ubuntu package? If so, which package? You should run "dpkg -S <path to link>" to check that. This link this not seem to be setup by libjna, so I wonder how you got it in your systems - from what I can tell it should *not* be there.

2) The fact that your library can't locate libjnidispatch.so in the default location
Ubuntu's libjna relies on 2 properties to locate the library: jna.boot.library.name and jna.boot.library.path. If not overriden jna.boot.library.name is set to "jnidispatch.system" and jna.boot.library.path is set to "/usr/lib/jni:/usr/lib/<multiarch path>/jni".
This second error seem to indicate that something is overriding either or both properties.

BTW, I'm setting this as incomplete again because there is still no clear reproducer for this issue. Your previous report of "1. the specific steps to reproduce this bug: Install package" do not reflect a correct reproducer for this - ie. installing the package works just fine. You need to provide information on: the code, the environment, and the exact commands being run.

[Frederik Hartmann (hartmafk)]

No problem!

Issue 1 could be the result of a band-aid fix to get the system working at all, so let's keep that back for the moment.

Regarding issue 2: I'm able to set up a fresh system with Jenkins on s390x.
What information would you need to be able to reproduce this issue?

[Tiago Stürmer Daitx (tdaitx)]

> Regarding issue 2: I'm able to set up a fresh system
> with Jenkins on s390x.
> What information would you need to be able to
> reproduce this issue?

I need to know what java application you are trying to run, where was it fetched from, how it was installed, and how it is being run. For each of this actions - and whatever else you can provide - I need to know the exact commands that are being run as well.

You mentioned Jenkins more than once, but it is unclear as to where it is a problem or not: you said you are able to setup a fresh system with it, which reads as it is working fine.

Also there is no official Jenkins package from Ubuntu - although there is a snap. There are many ways to get it installed from upstream as well as many different versions.

Without step by step instructions I can't reproduce this issue - I mean, I tried with a few jna examples I found online, they all worked. At the same time these examples didn't set either jna.boot.library.name or jna.boot.library.path, sticking to the default values for both properties. As I said earlier it seems to me that the application you are trying to run is setting one or both of these properties according to the error you are reporting.

[Frederik Hartmann (hartmafk)]

I'm trying to run Jenkins installed from the official Jenkins repository.

SO far no Jenkins installation has worked out of the box and my mention that I can install a fresh system was an offering to you to replay the installation and make sure that no changes remain and I have the full installation trace.

Here is what I did on a fresh 18.04 system(copy from https://jenkins.io/doc/book/installing/#debian-ubuntu):

apt install openjdk-8-jre-headless
apt install libjna-java

wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

After installation and initial configuration the url

http://xxx:8080/restart

will give the following message:
Jenkins cannot restart itself as currently configured.

If the following link is added, everything works:

/usr/lib/s390x-linux-gnu/libjnidispatch.so -> /usr/lib/s390x-linux-gnu/jni/libjnidispatch.system.so

It is important to note that some/most parts of jna can still work with jni being unavailable, it is only required for some of the functionality.

Similar issues (with a different program) have been discussed here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858876

[Tiago Stürmer Daitx (tdaitx)]

I installed jenkins 2.138.1 from http://pkg.jenkins.io/debian-stable and while I was unable to reproduce the issue. I looked at its structure and found out that they package their own jna.jar with its own libjnidispatch.so libraries inside.

Their jna jar file is located at ./WEB-INF/lib/jna-4.5.2.jar inside the jenkins.war file. The jna-4.5.2.jar contains libraries for quite a few classes, including com/sun/jna/linux-s390x/libjnidispatch.so.

This means that one is not required to install libjna-java as jenkins is expected to use its own jna jar file.

That said, in order to debug why loading is failing you can update the JAVA_ARGS in /etc/default/jenkins to
# Allow graphs etc. to work even when an X server is present
JAVA_ARGS="-Djava.awt.headless=true -Djna.debug_load.jna=true -Djna.debug_load=true -XshowSettings"

Then follow these steps:
1. restart jenkins (/etc/init.d/jenkins restart)
2. login into the web interface
3. look for "jni" and "jnidispatch" in /var/log/jenkins/jenkins.log

In my case the output was:
Trying (via loadLibrary) jnidispatch
Looking in classpath from WebAppClassLoader=Jenkins v2.138.1@2a2d45ba for /com/sun/jna/linux-x86-64/libjnidispatch.so
Found library resource at jar:file:/var/cache/jenkins/war/WEB-INF/lib/jna-4.5.2.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
Trying /tmp/jna--1712433994/jna5981117083713165017.tmp
Found jnidispatch at /tmp/jna--1712433994/jna5981117083713165017.tmp

Which means that, even with the libjna-java/libjna-jni installed, it is still loading jna's internal classes and libraries from the jenkins war file. I did re-test with libjna-java/libjna-jni uninstalled and the results are - as expected - the same.

Take a look at the jenkins log and check what your system is trying to do. The log should also contain various java properties (due to the -XshowSettings argument), so you can look and check whether any of the jna properties are being overridden.

For completion's sake here are my results from the provided test case:

> After installation and initial configuration the url
> http://xxx:8080/restart

After installing jenkins deb I accessed its web interface at port :8080, used the first time password (as indicated by the log), selected the default configuration/plugins (test case was not clear as to what option I should use), waited for it to finish installation, got asked to configure a user, then got logged in as admin. There was never a request to restart jenkins nor a indication from jenkins that I should it.

Just in case, I used the :8080/restart and it successfully restarted jenkins.

> will give the following message:
> Jenkins cannot restart itself as currently configured.

After restart there were no similar messages in either the logs or the web interface.

Two points that are not clear whether they are the same as they were not explicitly informed in the test case:
- the jenkins version being used, as a different version might pack some other jna or configure it differently
- the configuration selected in the web interface, as installing different plugins might cause some other issue to pop-up (eg. plugin packs its own jna, overrides some configuration, etc).

[Dimitri John Ledkov (xnox)]

Is this on Cosmic? Bionic? or Xenial?

I see that system.so is used everywhere by default, on things that do intentionally use system jna.

However, on xenial, there is one patch missing to ignore the override properties.

Jenkins as shipped in the 3rd party repo does not use system deb:libjna-java and does not need it installed at all.

[Frederik Hartmann (hartmafk)]

Tiago, thanks for having a look!

It seems that this issue was fixed on 1.136 (https://jenkins.io/changelog/)
The s390x libs haven't been included in the shipped library prior to, a system install was required at that time.

Anyhow, the locally installed libraries should be available and in the path.
Otherwise, changing the location/adding a link wouldn't have fixe the issue.

[Dimitri John Ledkov (xnox)]

I installed jenkins using https://jenkins.io/doc/book/installing/#debian-ubuntu on an s390x Ubuntu cosmic, using openjdk-8-jre-headless, without libjna-java or libjna-jni installed on the system as neither are required.

[Frederik Hartmann (hartmafk)]

I have verified the fix with the latest stable jenkins release.
It was in fact changed and local libjni/jna is no longer required.

Since the bug is fixed the only remaining question is if the handing of the system.so approach is a good idea and I think that this might not be the best place for this discussion.

I think that closing this report would be appropriate.

[Tiago Stürmer Daitx (tdaitx)]

> It seems that this issue was fixed on 1.136 (https://jenkins.io/changelog/)
> The s390x libs haven't been included in the shipped library prior to, a system install was required at that time.

Right, that would explain why it would not work prior to that.

> Anyhow, the locally installed libraries should be available and in the path.
> Otherwise, changing the location/adding a link wouldn't have fixe the issue.

The libraries are available in the path, only not with the default
name, and that is intentional. The libjna packages in Debian/Ubuntu
are designed to:
1. be used by application that don't embed their own jna jar file
2. prevent Debian/Ubuntu's libjnidispatch library from being loaded
when the application provides its own jna jar file, otherwise there's
a high chance of a version conflict

The above reasons is why the library is called
"libjnidispatch.system.so" instead of "libjnidispatch.so". The main
discussion on this is, as you pointed, in Debian bug
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858876

Any third party including their own jna.jar is responsible for
supporting it, from the java class to the arch native libraries.

Jenkins could improve on this by replacing embedded jars and libraries
with the ones packaged on Debian/Ubuntu, specially on the cases of
native libraries. Or, should they prefer to keep embedding the native
libraries, their package should move from "Architecture: all" to only
declaring the supported architectures.

[Tiago Stürmer Daitx (tdaitx)]

> I have verified the fix with the latest stable jenkins release.
> It was in fact changed and local libjni/jna is no longer required.

Great, thanks for the feedback.

> Since the bug is fixed the only remaining question is if the handing
> of the system.so approach is a good idea and I think that this might
> not be the best place for this discussion.

The debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858876 has more visibility and an actual discussion going on about the system library rename, including pros and cons.

> I think that closing this report would be appropriate.

I have updated the status to Invalid.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-09-20 07:00 EDT-------
IBM Bugzilla status -> closed; Also closed by Canonical