Add support for Debian 9 Stretch

[Ben Shum]

Evergreen master / OpenSRF master

Both Evergreen and OpenSRF need to have new makefile targets and instructions for setting up on Debian 9 Stretch distribution.

Branches and details to come next.

[Ben Shum (bshum)]

Initial branches here:

OpenSRF: user/bshum/debian9

http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/bshum/debian9

Evergreen: user/bshum/debian9

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/bshum/debian9

[Ben Shum (bshum)]

Some notes about the process so far for me:

- ldconfig did not appear to be working correctly for me, where we setup the /openils/lib shared library in /etc/ld.so.conf.d/opensrf.conf ; I get errors in startup and many services fail to run correctly. The workaround I found is to set the LD_LIBRARY_PATH manually in opensrf user's .bashrc:

export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/openils/lib

- I have to retest the manual installation of nodejs (npm package is deprecated and removed from Stretch, so we have to try manually installing nodejs). I copied the style we used for Debian Wheezy and Ubuntu Trusty.

[Ben Shum (bshum)]

Also a note about websockets setup. This message gets displayed when you go to create the websockets instance:

systemd is in use, no init script installed
use the '<email address hidden>' service to control your new instance
sample commands:
systemctl start <email address hidden>
systemctl enable <email address hidden>

With systemd, it looks like you need to use those sample commands to enable/start/stop the service name.

So, using "systemctl enable <email address hidden>" adds it as a service that can be stopped, started, etc.

[Mike Rylander (mrylander)]

Ben, do you happen to have the errors you mention in comment #2? I've found some apocryphal stories that somehow file capabilities or systemd or setuid executables are involved.

[Mike Rylander (mrylander)]

So, what's happening here is that, for some reason, a modern ldconfig is not finding libraries that do not start with "lib". But it gets stranger...

After symlinking liboils_cstore.so to oils_cstore.so.2.0.0, ldconfig finds and caches an entry for oils_cstore.so.2! At this point, if you adjust the <implementation> for cstore to read oils_cstore.so.2, it works.

Looking into why this is, and if/how we can convince ldconfig to do the right thing...

[Mike Rylander (mrylander)]

More:

The main difference in opensrf-c is that, on Wheezy (and, I'll guess, other currently-working distros) libtool sets the RPATH to /openils/lib, but on Stretch, it sets RUNPATH. The difference is that RPATH is the first thing that's evaluated when looking for a library, before LD_LIBRARY_PATH and ld.so.cache, but RUNPATH is (supposed to be) evaluated between the LD_LIBRARY_PATH and ld.so.cache. Per: https://www.physics.drexel.edu/~wking/unfolding-disasters-old/posts/rpath/

Still looking into if/why this matters, but it looks like a good lead right now. We may just need to force libtool to set RPATH.

Also, per http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html ldconfig does expect files it will investigate as libraries to start with "lib" which explains why the symlink in my previous comment allows things to work (almost).

[Mike Rylander (mrylander)]

More:

I can't yet figure out how to tell libtool to set RPATH instead of RUNPATH on Stretch.

The following patch:

diff --git a/src/libopensrf/Makefile.am b/src/libopensrf/Makefile.am
index fd3729b9..9bd043d1 100644
--- a/src/libopensrf/Makefile.am
+++ b/src/libopensrf/Makefile.am
@@ -98,6 +98,7 @@ noinst_PROGRAMS = osrf_json_test
 bin_PROGRAMS = opensrf-c
 opensrf_c_SOURCES = opensrf.c
 opensrf_c_DEPENDENCIES = libopensrf.la
+opensrf_c_LDFLAGS = -rpath $(libdir)

 osrf_json_test_SOURCES = osrf_json_test.c $(JSON_TARGS) $(JSON_DEP) $(JSON_TARGS_HEADS) $(JSON_DEP_HEADS)
 osrf_json_test_DEPENDENCIES = libopensrf.la

Seems to generate the right output during the build:

/bin/bash ../../libtool --tag=CC --mode=link gcc -D_LARGEFILE64_SOURCE -DNDEBUG -pipe -g -Wall -O2 -fPIC -I/home/opensrf/opensrf/include/ -I/usr/include/libxml2/ -I/usr/include/apache2 -I/usr/include/apr-1.0 -DASSUME_STATELESS -DOSRF_STRICT_PARAMS -rdynamic -fno-strict-aliasing -DOSRF_JSON_ENABLE_XML_UTILS -g -O2 -rpath /openils/lib -o opensrf-c opensrf.o -lopensrf -lxml2 -lreadline -lncurses -ldl
libtool: link: gcc -D_LARGEFILE64_SOURCE -DNDEBUG -pipe -g -Wall -O2 -fPIC -I/home/opensrf/opensrf/include/ -I/usr/include/libxml2/ -I/usr/include/apache2 -I/usr/include/apr-1.0 -DASSUME_STATELESS -DOSRF_STRICT_PARAMS -rdynamic -fno-strict-aliasing -DOSRF_JSON_ENABLE_XML_UTILS -g -O2 -o .libs/opensrf-c opensrf.o /home/opensrf/opensrf/src/libopensrf/.libs/libopensrf.so -lxml2 -lreadline -lncurses -ldl -Wl,-rpath -Wl,/openils/lib

But readelf still says:

 0x000000000000001d (RUNPATH) Library runpath: [/openils/lib]

Instead of what I want:

 0x000000000000000f (RPATH) Library rpath: [/openils/lib]

Thoughts appreciated...

[Jason Stephenson (jstephenson)]

Rather than mess with rpath, which is fragile and not necessarily portable, I think we should just let libtool do its thing and name the libraries properly. It is meant to solve these kinds of problems, after all.

[Mike Rylander (mrylander)]

I don't disagree, Jason. I think that does put stretch support outside the Evergreen 3.0 window, though, and into 3.1 territory.

It will mean changing the opensrf.xml file for C-based app implementations in Evergreen, and will almost certainly break binary compatibility at the OpenSRF level. I'm a'feared of doing all that this late in the release cycle. My hope was to get stretch support now, without needing an OpenSRF version bump...

Should we target this at Evergreen 3.next and (the hypothetical) OpenSRF 2.next?

[Galen Charlton (gmc)]

Well, bumping up to an OpenSRF 2.6.0 for the purpose of naming the libraries as libtool expects is reasonable on the OpenSRF side, and (although the timing is... not great) I'd rather try for Stretch support in 3.0, even if it doesn't make it in until closer to the RC. Folks are going to have to touch opensrf.xml anyway when upgrading to 3.0 because of qstore.

[Jason Stephenson (jstephenson)]

This is interesting: I am building a Debian 9 VM right now. I usually create the OpenSRF user (opensrf) during installation. When I did so today with Debian 9 I get the following error:

The username you entered (opensrf) is reserved for use by the system. Please select a different one.

I can find nothing in the release notes about this user being added or why.

Searching the Debian site for OpenSRF turns up this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637391

I don't think the user would have been added because of a request to package OpenSRF.

Searching the wiki and the packages turns up nothing.

[Jason Stephenson (jstephenson)]

Responding to Ben's comment #2:

ldconfig - I think part of this is explained indirectly in the release notes.

All executables are compiled with position independent code by GCC 6, even static libraries. I surmise that this makes it more important for libraries to follow standard naming conventions:

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#pie-is-now-default

It could be that we need to modify C library code to accommodate PIC, though I doubt it.

Node.js - The status of Node.js packages is also explained in the release notes. They are deprecated because of security issues with the V8 JavaScript engine and the lack of maintainers for the packages.

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#libv8

There are additional changes that may have an impact on Evergreen and Evergreen users, and I highly recommend that anyone thinking of moving to Debian 9 read the release notes.

[Jason Stephenson (jstephenson)]

RE the opensrf user.

Once the VM is created, the opensrf user does not exist.

[Jason Stephenson (jstephenson)]

After you add the opensrf user, it has a normal user id: 1001. Not a system user id. This seems like a bug in the installer.

[Jason Stephenson (jstephenson)]

Changing the library names in OpenSRF has worked for me. I'm moving on to the Evergreen side of things. I'll share my branches later today.

I think we should also remove Debian Wheezy support or is the consensus that should be its own bug?

[Galen Charlton (gmc)]

> Changing the library names in OpenSRF has worked for me. I'm moving on
> to the Evergreen side of things. I'll share my branches later today.

Great!

> I think we should also remove Debian Wheezy support or is the consensus
> that should be its own bug?

I think it should be a separate bug.

[Jason Stephenson (jstephenson)]

I have pushed my signoff branches to working with an extra commit to rename the C libraries for both OpenSRF and Evergreen:

http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/user/dyrcona/lp1708048_debian9

and

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dyrcona/lp1708048_debian9

I am able to start OpenSRF services, login via srfsh and the web staff client, and successfully ran a search. make check passes.

[Jason Stephenson (jstephenson)]

I should add that all live tests succeed on Debian 9.

We should test this on Ubuntu and Debian 8 before it goes into master, IMHO.

[Ben Shum (bshum)]

Tested both OpenSRF and Evergreen branches with a fresh Ubuntu 16.04 build and it seemed to run properly for me. I guess we'll need to test it with an upgrade too.

[Galen Charlton (gmc)]

Works for me with a fresh install on Debian Jessie.

[Galen Charlton (gmc)]

And works for me on Debian Stretch. I suspect there's an opportunity to reduce the number of Perl dependencies that are brought in from CPAN, but that can be worked out between now and the release.

[Galen Charlton (gmc)]

And not that it's a particularly long-term concern, also works for me on Wheezy.

[Galen Charlton (gmc)]

So, I'm satisfied that we'll be OK for Evergreen 3.0-beta2 and OpenSRF next-alpha, so I'll plan on merging. However, if somebody could test on Trusty, that would be groovy.

[Jason Stephenson (jstephenson)]

I can test on trusty today.

[Jason Stephenson (jstephenson)]

I tested on trusty.

make check succeeds for both Evergreen and OpenSRF.

All Evergreen live tests pass with this branch.

[Galen Charlton (gmc)]

Pushed to OpenSRF and Evergreen master. Thanks, Ben and Jason!