[Ubuntu 22.04] smc_run does not work, libsmc-preload.so cannot be preloaded

[bugproxy]

SRU Justification:
==================

[ Impact ]

 * The smc_run tools does not work since libsmc-preload.so is placed
   in the wrong folder (/usr/lib64/ instead of /usr/lib/).

 * smc_run ping <ip>
   ERROR: ld.so: object 'libsmc-preload.so' from LD_PRELOAD cannot be preloaded
   (cannot open shared object file): ignored.
   PING 172.18.39.17 (172.18.39.17) 56(84) bytes of data.

[ Test Plan ]

 * Setup an Ubuntu on s390x system with access to RoCE Express adapters.

 * Setup a network for RoCE (RDMA).

 * Use smc_run to ding a different adapter
   (can also be done from one port to a second port of the same adapter)
   smc_run ping <ip>

[ Where problems could occur ]

 * A change in the Makefile is sufficient.

 * This is based on an upstream commit
   and affects Ubuntu only.

 * Problems could occur in case people adapted the wrong path
   in their code/scripts manually.

[ Fix ]

 * Add lsb-release as Build-Depends to debian/control, since it's not
   installed by default in all build environments, but the lsb_release
   command is used in the Makefile.

 * Patch the Makefile with upstream commit
   '[PATCH] adjust default library path for Ubuntu'
   46f397208fa0001f3f14e63f01c842c79ab3c6d6

[ Other Info ]

 * Affects focal and jammy.
__________

---Problem Description---

[Ubuntu 22.04] smc_run does not work, libsmc-preload.so cannot be preloaded
===========================================================================

The reason for this is that the aforementioned shared object is installed under: /usr/lib64/libsmc-preload.so

Which is AFAICT not what we want, because AFAIU it violates the Debian Policy Manual (https://www.debian.org/doc/debian-policy/ch-opersys.html#s-fhs), and ld.so in Ubuntu has a default search path that understandably does not cover /usr/lib64.

Interestingly the upstream Make file does try to detect if we are run on Ubuntu, and choose the install path accordingly.

Hereby it relies on "lsb_release -si" returning "Ubuntu", which may or may not be a good idea -- I don't know.

In any case if the upstream Makefile does not suit Ubuntu/Debian, then the packager is ought to patch it.

Thus in my opinion this is a packaging problem.

Contact Information = <email address hidden>

---uname output---
Linux t3545018.lnxne.boe 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:30:43 UTC 2023 s390x s390x s390x GNU/Linux

Machine Type = LPAR

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 $ smc_run ping 172.18.39.17
ERROR: ld.so: object 'libsmc-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
PING 172.18.39.17 (172.18.39.17) 56(84) bytes of data.

BTW
$ LD_PRELOAD=libsmc-preload.so LD_DEBUG=libs LD_LIBRARY_PATH=/usr/lib64/ ping 172.18.39.17
does work as expected.

Userspace tool common name: smc_run

The userspace tool has the following bit modes: na

Userspace rpm: smc-tools_1.7.0-0ubuntu1_s390x.deb

Userspace tool obtained from project website: na

*Additional Instructions for <email address hidden>:
-Attach ltrace and strace of userspace application.

[Frank Heimes (fheimes)]

Looks like upstream assumes that the default library path is lib64 for s390x - which is the case for I think RH and SLES, but on Debian and Ubuntu lib is the default location for 64-bit s309x libraries.

I remember vaguely that we had to fix it already in the past ?! maybe in a different package...

But ideally upstream should differentiate and check for the different distros - let's see what the best approach is ...

[Frank Heimes (fheimes)]

Hello Halil, so this is caused by a missing (build) dependency of a tool that is used in the Makefile (lsb_release).
If it's in place 'libsmc-preload.so' will be installed on Ubuntu at '/usr/lib/s390-linux-gnu/libsmc-preload.so' - according to the upstream Makefile code (so not in /usr/lib).
Should it be in /usr/lib instead or at least sym-linked to /usr/lib?
If not, it's okay with the fixed build dependency, if not I need to tweak the packaging (or even the Makefile).

[Frank Heimes (fheimes)]

Fixed version got build in PPA (for all major architectures) for further testing:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2009495

[Frank Heimes (fheimes)]

debdiffs for the affected series

[Frank Heimes (fheimes)]

Uploaded for Lunar first ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package smc-tools - 1.8.2-0ubuntu2

---------------
smc-tools (1.8.2-0ubuntu2) lunar; urgency=medium

  * Add lsb-release as Build-Depends to debian/control, since it's not
    installed by default in all build environments, but the lsb_release
    command is used in the Makefile. If it's not there the Makefile defaults
    to the the wrong folder /usr/lib64 for Ubuntu. LP: #2009495
  * Add debian/links file to ensure that libsmc-preload.so is also
    accessible from /usr/lib (and not only from /usr/lib/s390x-linux-gnu).

 -- Frank Heimes <email address hidden> Wed, 08 Mar 2023 18:58:21 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-03-15 13:41 EDT-------
(In reply to comment #7)
> Hello Halil, so this is caused by a missing (build) dependency of a tool
> that is used in the Makefile (lsb_release).
> If it's in place 'libsmc-preload.so' will be installed on Ubuntu at
> '/usr/lib/s390-linux-gnu/libsmc-preload.so' - according to the upstream
> Makefile code (so not in /usr/lib).
> Should it be in /usr/lib instead or at least sym-linked to /usr/lib?
> If not, it's okay with the fixed build dependency, if not I need to tweak
> the packaging (or even the Makefile).

Hello Frank!

Sorry I have no direct answer for your question. My understanding of this multiarch stuff is not sufficient.

In what cases are shared libraries meant to be in /usr/lib or symlinked to it? I had a quick stab at the corresponding documentation, but I failed to figure it out. Do libraries that have the same architecture as the dpkg --print-architecture reports have to go under /usr/lib/<triplet> or rather under /usr/lib (in which case only the foreign architecture shared libraries would go under /usr/lib/<triplet>). Based on the specs I have read I lean towards the former, but I'm far from confident, and your question makes me ask myself if it is actually the latter.

[Frank Heimes (fheimes)]

Ah, no worries /usr/lib/s390-linux-gnu is used here by default for arch specific lib (there are several in the s390-tools package, too),
but I've now created a symlink in the packaging to just /usr/lib/.
So everyone should be fine either way.

You can try it out if you want with this one:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2009495/+packages
https://launchpad.net/~fheimes/+archive/ubuntu/lp2009495/+files/smc-tools_1.7.0-0ubuntu1.22.04.1_s390x.deb

[Steve Langasek (vorlon)]

The purpose of the /usr/lib/s390x-linux-gnu directory is to make libraries for multiple architectures co-installable on the same filesystem (multiarch).

If you're going to create a symlink from /usr/lib, and the package in question can't be co-installable because it also ships files in /usr/bin, and the programs that use it only access it via the /usr/lib/ path instead of the /usr/lib/s390x-linux-gnu path *anyway*, then it's useless to install the library in /usr/lib/s390x-linux-gnu at all. Instead of adding a link, you should *move* the file at build-time to the /usr/lib directory where it will be looked for.

Also, the filename ("preload") implies this isn't actually a shared library that programs will link against, but that they preload/dlopen it instead. For such cases, it's preferred that the library be shipped in a per-project subdirectory of /usr/lib. See libfakeroot for an example of this. This is not a blocker for SRU, just a comment on best-practices here.

The first issue, however, is an SRU reject for me. If we're going to touch this in an SRU, then we should do it the right way, not with extraneous extra symlinks.

[Steve Langasek (vorlon)]

An upload of smc-tools to kinetic-proposed has been rejected from the upload queue for the following reason: "Per comments on bug report".

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-03-17 21:01 EDT-------
(In reply to comment #15)
> The purpose of the /usr/lib/s390x-linux-gnu directory is to make libraries
> for multiple architectures co-installable on the same filesystem (multiarch).
>
> If you're going to create a symlink from /usr/lib, and the package in
> question can't be co-installable because it also ships files in /usr/bin,
> and the programs that use it only access it via the /usr/lib/ path instead
> of the /usr/lib/s390x-linux-gnu path *anyway*, then it's useless to install
> the library in /usr/lib/s390x-linux-gnu at all. Instead of adding a link,
> you should *move* the file at build-time to the /usr/lib directory where it
> will be looked for.
>
> Also, the filename ("preload") implies this isn't actually a shared library
> that programs will link against, but that they preload/dlopen it instead.
> For such cases, it's preferred that the library be shipped in a per-project
> subdirectory of /usr/lib. See libfakeroot for an example of this. This is
> not a blocker for SRU, just a comment on best-practices here.
>
> The first issue, however, is an SRU reject for me. If we're going to touch
> this in an SRU, then we should do it the right way, not with extraneous
> extra symlinks.
>
> An upload of smc-tools to kinetic-proposed has been rejected from the upload
> queue for the following reason: "Per comments on bug report".

Regarding the second point, AFAIU the smc_run is just a convenience wrapper,
and it is a documented than one can also "smc-fy" applications (that are not installed as a part of the smc-tools package) by invoking them like this:
#LD_PRELOAD=libsmc-preload.so <application_start_cmd>
(for reference have a look at https://www.ibm.com/docs/en/linux-on-systems?topic=sps-setup-4).

My guess is that the application needs to be a 64-bit one although the aforementioned documentation does not seem to care to state it.

In any case, I don't think we would be happy with some "per project subdirectory of /usr/lib" because of that piece of documentation.

Regarding the first part, I don't quite understand enough. I'm quite confused actually. AFAIU for supporting the "smcifying" both 64 and 31 bit programs, we would have to provide both an 64 and a 31 bit version of libsmc-preload.so. Is that right?

If that is right, my guess is that the 64 bit variant would go under
/usr/lib/s390x-linux-gnu
and the 31 bit variant would go under
/usr/lib/s390-linux-gnu
and I hope ld.so would pick up the "right one". Does that sound about right? It does not mesh well with this has to go under /usr/lib idea, so I guess there is somewhere a mistake in my train of thought. Can you help me to figure it out?

Regards,
Halil

[Frank Heimes (fheimes)]

@Steve, thanks for the details. Well, I actually wanted to improve the ease of use with the link, since I saw it like this in other environments, but have missed the consequence you kindly described.

@Halil please notice that we don't have support for any 31-bit code (s390) in Ubuntu at all.
So we will have to care for Z in Ubuntu on /usr/lib/s390x-linux-gnu only.

I think it's indeed best to look for a way to *move* the file at build-time to /usr/lib (like suggested). (But I'll also have a look at libfakeroot first.)

[Frank Heimes (fheimes)]

I requested a general change upstream for the default Ubuntu library path:
smc-tools PR #6 - "adjust default library path for Ubuntu"
https://github.com/ibm-s390-linux/smc-tools/pull/6

[Utkarsh Gupta (utkarsh)]

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

[Asa Mirzaieva (asaly12)]

I created a quilt patch and added lsb-release to the Build-Depends to fix this issue. Following the behavior or noble, the error still occurs when running as a non-root user. I also did a test build in a PPA for jammy: https://launchpad.net/~asaly12/+archive/ubuntu/smc-tools-sru/+build/27808950

[Frank Heimes (fheimes)]

Thank you Asa for the work on this SRU for jammy!
- no old patches to care about, no new ones
- debdiff looks reasonable
- lintian (-EvIL -pedantic) on src and DEBs looks reasonable
- build log looks good, no issues
- sanity checks are fine, install and upgrade tests tests are fine too
- LP bug public
- changelog:
   - entries could be a bit more detailed
   - LP bug reference needed
   - correct version string
   - correct code-name
   - maintainer filed ok

Looks good with these changes - I'm sponsoring and uploading ...

[Frank Heimes (fheimes)]

Uploaded slightly modified version (changelog only) based on:
https://launchpad.net/~asaly12/+archive/ubuntu/smc-tools-sru/+build/27808950

Now waiting for approval (by the SRU team).

[bugproxy (bugproxy)]

looks good with correct version string, code-name and changes (incl. LP bug reference), maintainer filed ok
Looks all pretty good, nothing to comment, so r

[Frank Heimes (fheimes)]

<Please ignore comment #18, which is due to an issue with the LP-BZ-bridge.>

[Mauricio Faria de Oliveira (mfo)]

Accepting for Jammy.
Observations:

0) Steve's comment #9 is addressed.

1) This change modify the location of a shipped/packaged library
*in an SRU*, but this looks OK *in this particular case* because:

A) one program in the package does not work by default (fixed with this)
B) only this package uses this library (no revdeps; low regression risk)

 $ dpkg-deb -x smc-tools_1.8.3-0ubuntu1_amd64.deb deb
 $ grep -rF libsmc-preload.so deb/
 deb/usr/bin/smc_run:LIB_NAME="libsmc-preload.so"
 grep: deb/usr/lib/libsmc-preload.so: binary file matches

 $ reverse-depends smc-tools
 No reverse dependencies found

2) Mantic/Noble contain the changes.

 $ rmadisrc smc-tools
  smc-tools | 1.2.2-0ubuntu1 | focal/universe | source
  smc-tools | 1.7.0-0ubuntu1 | jammy/universe | source
  smc-tools | 1.8.3-0ubuntu1 | mantic/universe | source
  smc-tools | 1.8.3-0ubuntu1 | noble/universe | source

 $ grep Build-Depends: smc-tools-*/debian/control
 smc-tools-1.7.0/debian/control:Build-Depends: debhelper-compat (= 12), libnl-genl-3-dev, bash-completion, pkg-config, lsb-release
 smc-tools-1.8.3/debian/control:Build-Depends: debhelper-compat (= 12), libnl-genl-3-dev, bash-completion, pkg-config, lsb-release

 $ grep -B4 lib64 smc-tools-*/Makefile
 smc-tools-1.7.0/Makefile-ifeq ($(ARCH),64)
 smc-tools-1.7.0/Makefile-ifeq ($(DISTRO),Ubuntu)
 smc-tools-1.7.0/Makefile-LIBDIR = ${PREFIX}/lib
 smc-tools-1.7.0/Makefile-else
 smc-tools-1.7.0/Makefile:LIBDIR = ${PREFIX}/lib64
 --
 smc-tools-1.8.3/Makefile-ifeq ($(ARCH),64)
 smc-tools-1.8.3/Makefile-ifeq ($(DISTRO),Ubuntu)
 smc-tools-1.8.3/Makefile-LIBDIR = ${PREFIX}/lib
 smc-tools-1.8.3/Makefile-else
 smc-tools-1.8.3/Makefile:LIBDIR = ${PREFIX}/lib64

[Mauricio Faria de Oliveira (mfo)]

Hello bugproxy, or anyone else affected,

Accepted smc-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/smc-tools/1.7.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Asa Mirzaieva (asaly12)]

Verified the -proposed package:
1. Installed smc-tools-1.7.0-0ubuntu1.1 from jammy-proposed (see log)

2. Run smc_run without sudo:
$ smc_run ping ubuntu.com
ERROR: ld.so: object 'libsmc-preload.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
PING ubuntu.com (185.125.190.21) 56(84) bytes of data.
^C
--- ubuntu.com ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5226ms

3. Run smc_run with sudo:
$ sudo smc_run ping ubuntu.com
PING ubuntu.com (185.125.190.20) 56(84) bytes of data.
^C
--- ubuntu.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3124ms

[Asa Mirzaieva (asaly12)]

apt install log:

[Mauricio Faria de Oliveira (mfo)]

ppc64el failed to build, no build logs; retrying.

[Mauricio Faria de Oliveira (mfo)]

ppc64el build finished successfully; all good.

[Launchpad Janitor (janitor)]

This bug was fixed in the package smc-tools - 1.7.0-0ubuntu1.1

---------------
smc-tools (1.7.0-0ubuntu1.1) jammy; urgency=medium

  * Add d/p/fix-library-path.patch to fix library path issue on Ubuntu,
    solves LP: #2009495
  * Add 'lsb_release' to Build-Depends in d/control to solve lintian warning.

 -- Asal Mirzaieva <email address hidden> Wed, 07 Feb 2024 15:36:29 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for smc-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Asa Mirzaieva (asaly12)]

Focal package already places libsmc_preload.so in /lib:

On Focal system:
$ apt download smc-tools
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/universe s390x smc-tools s390x 1.2.2-0ubuntu1 [26.0 kB]
Fetched 26.0 kB in 0s (85.6 kB/s)
$ dpkg -c smc-tools_1.2.2-0ubuntu1_s390x.deb | grep libsmc
-rw-r--r-- root/root 6216 2020-02-10 01:42 ./usr/lib/s390x-linux-gnu/libsmc-preload.so

Closing focal issue as Invalid.