perl backend can't use dynamically loaded modules (DBI, POSIX...)

Bug #90812 reported by Roberto Maurizzi
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openldap
Invalid
Undecided
Unassigned
openldap (Debian)
Fix Released
Unknown
openldap (Ubuntu)
Fix Released
Medium
Unassigned
openldap2.2 (Ubuntu)
Invalid
Medium
Unassigned
openldap2.3 (Ubuntu)
Invalid
Medium
Unassigned

Bug Description

In slapd back_perl backend if I "use" a module I get "unresolved symbol" errors.

I'm using slapd-2.2.26-5ubuntu3.1 on Edgy.

Everything works if I don't use modules. For example if in my module i "use DBI", running "slapd -d 0" I get:

slapd: symbol lookup error: /usr/lib/perl5/auto/DBI/DBI.so: undefined symbol: Perl_Tstack_sp_ptr

if I instead "use POSIX" as in the SampleLDAP.pm module given in the openldap sources, i get:

Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
 at /usr/lib/perl/5.8/POSIX.pm line 26

Searching for documentation, I've found this article http://www.samag.com/documents/s=9370/sam0505g/0505g.htm that stresses the importance that both openldap and perl are compiled with the same compiler version and options.

Am I missing something obvious, or there's been a misalignment in the releases of slapd and perl packages?

To reproduce the error:
define a perl backend "db" adding the following lines to the end of /etc/ldap/slapd.conf:

moduleload back_perl
database perl
suffix "dc=perl,dc=example,dc=com"
perlModulePath /etc/perl/SampleLDAP.pm
perlModule SampleLDAP

Create a file with the following lines in /etc/perl/SampleLDAP.pm

package SampleLDAP;
use POSIX;
sub new
{
        my $class = shift;
        my $this = {};
        bless $this, $class;
        print STDERR "Posix Var " . BUFSIZ . " and " . FILENAME_MAX . "\n";
        return $this;
}
sub init
{
        print STDERR "Here in init\n";
        return 0;
}
1;

Now you should start slapd:

sudo slapd

You'll get:

Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
  at /usr/lib/perl/5.8/POSIX.pm line 26
 Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
 BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
 Compilation failed in require at (eval 2) line 1.
 BEGIN failed--compilation aborted at (eval 2) line 1.

Using /etc/init.d/slapd start does NOT print the message because the output of start-stop-daemon is put in a $reason variable that's not printed, logged or evaluated (script bug?) but if you add echo $reason to the script you get the same error.

Changing the "use"d module changes the situation only slightly, for example substituting "use POSIX" with "use DBI" you get:

slapd: symbol lookup error: /usr/lib/perl5/auto/DBI/DBI.so: undefined symbol: Perl_Tstack_sp_ptr

I'd more than willing to perform more tests if you can suggest some.

Ciao,
    Roberto Maurizzi

Revision history for this message
In , magicfox (magicfox) wrote :

On Sun, Sep 11, 2005 at 10:13:53AM +0200, Marc Chantreux wrote:
> Package: slapd
> Version: 2.2.26-4
> Severity: normal
Hi. Same behaviour with 2.2.23-8 (Sarge)

> this message was produced by slapd while loading SampleLDAP.pm
provided in the
> openldap source tree:
(and after having copied the 4 sample lines in the comments on the top
of the module file to the slapd.conf configuration file)

> Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module
POSIX:
> [...]
Got the same here.

It looks like the compilation of upstream 2.2.30 version doesn't raise
these errors. What about its packaging for unstable?

Cheers,

--
Cyril Brulebois

Revision history for this message
Roberto Maurizzi (r-maurizzi) wrote :

In slapd back_perl backend if I "use" a module I get "unresolved symbol" errors.

I'm using slapd-2.2.26-5ubuntu3.1 on Edgy.

Everything works if I don't use modules. For example if in my module i "use DBI", running "slapd -d 0" I get:

slapd: symbol lookup error: /usr/lib/perl5/auto/DBI/DBI.so: undefined symbol: Perl_Tstack_sp_ptr

if I instead "use POSIX" as in the SampleLDAP.pm module given in the openldap sources, i get:

Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
 at /usr/lib/perl/5.8/POSIX.pm line 26

Searching for documentation, I've found this article http://www.samag.com/documents/s=9370/sam0505g/0505g.htm that stresses the importance that both openldap and perl are compiled with the same compiler version and options.

Am I missing something obvious, or there's been a misalignment in the releases of slapd and perl packages?

To reproduce the error:
define a perl backend "db" adding the following lines to the end of /etc/ldap/slapd.conf:

module_load back_perl
database perl
suffix "dc=perl,dc=example,dc=com"
perlModulePath /etc/perl/SampleLDAP.pm
perlModule SampleLDAP

Create a file with the following lines in /etc/perl/SampleLDAP.pm

package SampleLDAP;
use POSIX;
sub new
{
        my $class = shift;
        my $this = {};
        bless $this, $class;
        print STDERR "Posix Var " . BUFSIZ . " and " . FILENAME_MAX . "\n";
        return $this;
}
sub init
{
        print STDERR "Here in init\n";
        return 0;
}
1;

Now you should start slapd:

sudo slapd

You'll get:

Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
  at /usr/lib/perl/5.8/POSIX.pm line 26
 Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
 BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
 Compilation failed in require at (eval 2) line 1.
 BEGIN failed--compilation aborted at (eval 2) line 1.

Using /etc/init.d/slapd start does NOT print the message because the output of start-stop-daemon is put in a $reason variable that's not printed, logged or evaluated (script bug?) but if you add echo $reason to the script you get the same error.

Changing the "use"d module changes the situation only slightly, for example substituting "use POSIX" with "use DBI" you get:

slapd: symbol lookup error: /usr/lib/perl5/auto/DBI/DBI.so: undefined symbol: Perl_Tstack_sp_ptr

I'd more than willing to perform more tests if you can suggest some.

Ciao,
    Roberto Maurizzi

description: updated
Revision history for this message
Roberto Maurizzi (r-maurizzi) wrote :

I've just tested a rebuild of the openldap package with the 'current' version of the perl system, and the error is the same.
The error seems to be a real one, not a simple packaging problem...

Ciao,
   Roberto

Revision history for this message
Quanah (quanah-zimbra) wrote :

The problem in this bug is user configuration error. The perlModulePath directive is to point to the directory where the perl module resides, not to the perl module itself. Please see slapd-perl(5). You can also look at ITS#4995 in the OpenLDAP ITS system at http://www.openldap.org/its/

--Quanah
OpenLDAP QA Engineer

Revision history for this message
Roberto Maurizzi (r-maurizzi) wrote :

I've corrected pertModulePath to point to the directory where the module is, as opposed to the file of the module.
No changes: The module gets loaded with both configurations, and with both slapd gives the reported symbol lookup errors and die on the instruction that uses the dinamically loaded module.

Same applies for 2.3 as shipped in Ubuntu Feisty.

Ciao,
   Roberto Maurizzi

Mathias Gug (mathiaz)
Changed in openldap2.2:
importance: Undecided → Medium
status: New → Triaged
Changed in openldap2.2:
status: Unknown → New
Revision history for this message
lmhd (maylein) wrote :

I have the same problem.
Is there any solution in the meantime?

Revision history for this message
In , Russ Allbery (rra-debian) wrote : Perl slapd backend should now work again

Version: 2.3.35-1

I'm pretty sure this bug has been fixed for some time. There was a later
documentation confusion, but that's also been cleared up by this point.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Changed in openldap2.2:
status: New → Fix Released
Revision history for this message
In , Debbugs Internal Request (owner-bugs) wrote : Internal Control

# A New Hope
# A log time ago, in a galaxy far, far away
# something happened.
#
# Magically this resulted in the following
# action being taken, but this fake control
# message doesn't tell you why it happened
#
# The action:
# Bug archived.
thanks
# This fakemail brought to you by your local debbugs
# administrator

Revision history for this message
Roberto Maurizzi (r-maurizzi) wrote :

So... in debian it's "fix released", here triaged... any news?
I've been using slapd-sql instead of slapd-perl for the problem at hand, so no problem functionality-wise (on the contrary... I think it's a better solution).

But I must say I'm starting to see problems on how (un)responsive Ubuntu is to bugs/regression in general. I find it "funny" that important applications takes months to pick up released fixes from debian or from other Ubuntu versions (see bug #146145 for some laughs)

Revision history for this message
In , Stephen Gran (sgran) wrote : reopening

reopen 327585
found 327585 2.3.35-1
thanks

Hey,

This is in all likelihood not fixed (I am struggling with similar
issues in freeradius, and I can guarantee the same techniques
are not working there, and we both use the Debian ltdl). See
https://bugs.launchpad.net/openldap/+bug/90812 for more reports of it
being broken.

Cheers,
--
 -----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : <email address hidden> |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
 -----------------------------------------------------------------

Revision history for this message
In , Stephen Gran (sgran) wrote : t

unarchive 327585
reopen 327585
found 327585 2.3.35-1
thanks
--
 -----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : <email address hidden> |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
 -----------------------------------------------------------------

Revision history for this message
In , Russ Allbery (rra-debian) wrote : Re: [Pkg-openldap-devel] Bug#327585: reopening

Stephen Gran <email address hidden> writes:

> reopen 327585
> found 327585 2.3.35-1
> thanks
>
> This is in all likelihood not fixed (I am struggling with similar issues
> in freeradius, and I can guarantee the same techniques are not working
> there, and we both use the Debian ltdl). See
> https://bugs.launchpad.net/openldap/+bug/90812 for more reports of it
> being broken.

Well, I tested the Perl backend in slapd a while back and it worked for
me, so I really do think this bug is closed. That's why I closed it. If
it isn't working for you, could you please report the exact error messages
that you're getting against the current version of the OpenLDAP slapd
package?

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Revision history for this message
In , Stephen Gran (sgran) wrote :

This one time, at band camp, Russ Allbery said:
> Stephen Gran <email address hidden> writes:
>
> > reopen 327585
> > found 327585 2.3.35-1
> > thanks
> >
> > This is in all likelihood not fixed (I am struggling with similar issues
> > in freeradius, and I can guarantee the same techniques are not working
> > there, and we both use the Debian ltdl). See
> > https://bugs.launchpad.net/openldap/+bug/90812 for more reports of it
> > being broken.
>
> Well, I tested the Perl backend in slapd a while back and it worked for
> me, so I really do think this bug is closed. That's why I closed it. If
> it isn't working for you, could you please report the exact error messages
> that you're getting against the current version of the OpenLDAP slapd
> package?

steve@spartacus:~$ sudo /etc/init.d/slapd start
Starting OpenLDAP: slapd - failed:
Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
 at /usr/lib/perl/5.8/POSIX.pm line 26
Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
Compilation failed in require at (eval 2) line 1.
BEGIN failed--compilation aborted at (eval 2) line 1.

Can't call method "config" on an undefined value.
steve@spartacus:~$ dpkg -l slapd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=========================-=========================-==================================================================
ii slapd 2.4.7-4 OpenLDAP server (slapd)

This is with the configuration and module described at
https://bugs.launchpad.net/openldap/+bug/90812

This is a symbol visibility problem caused by dlopening an embedded
interpreter and using Debian's ltdl, which resolves symbols without
RTLD_GLOBAL.

Thanks,
--
 -----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : <email address hidden> |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
 -----------------------------------------------------------------

Revision history for this message
In , Russ Allbery (rra-debian) wrote :

retitle 327585 slapd: [i386] perl backend failed to load XS (unknown symbols)
thanks

Stephen Gran <email address hidden> writes:

> steve@spartacus:~$ sudo /etc/init.d/slapd start
> Starting OpenLDAP: slapd - failed:
> Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
> at /usr/lib/perl/5.8/POSIX.pm line 26
> Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
> BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
> Compilation failed in require at (eval 2) line 1.
> BEGIN failed--compilation aborted at (eval 2) line 1.
>
> Can't call method "config" on an undefined value.

Okay, thanks.

I could have sworn that this wasn't happening with 2.3.30 or
thereabouts... oh, I know what the difference is. I was testing on
amd64. Right, now I understand.

This is that problem that's caused by Perl modules not being linked with
libperl on i386 (but not on any other architecture) and instead
duplicating the symbols between perl and libperl and leaving them
undefined in the modules. This works correctly if the modules are linked
with libperl, which they are on amd64 and Debian's other architectures.

You're right, this isn't closed; it's just i386-specific.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Revision history for this message
In , Steve Langasek (vorlon) wrote : Re: [Pkg-openldap-devel] Bug#327585: Bug#327585: reopening
Download full text (3.2 KiB)

On Sun, Feb 03, 2008 at 05:17:45PM -0800, Russ Allbery wrote:
> Stephen Gran <email address hidden> writes:

> > reopen 327585
> > found 327585 2.3.35-1
> > thanks

> > This is in all likelihood not fixed (I am struggling with similar issues
> > in freeradius, and I can guarantee the same techniques are not working
> > there, and we both use the Debian ltdl). See
> > https://bugs.launchpad.net/openldap/+bug/90812 for more reports of it
> > being broken.

> Well, I tested the Perl backend in slapd a while back and it worked for
> me, so I really do think this bug is closed. That's why I closed it.

Did you test the perl backend using any XS modules?

The Debian OpenLDAP packages are linked against the system libltdl, which
opens all objects with RLTD_LOCAL instead of RTLD_GLOBAL. RTLD_GLOBAL is
what upstream libltdl uses, but it's not a sane behavior for a
general-purpose system library because it's completely unscalable and has
resulted in symbol clobbering bugs in the past; so the Debian package is
patched to use RLTD_LOCAL, which in turn is not usable for the case of
multiple levels of plugins where the plugins at the bottom level reference
symbols that they expect to be provided by the caller.

Since the perl package in Debian is not linked against libperl, perl
extensions also cannot be linked against libperl since this would cause two
copies of the perl XS symbols to be loaded. So perl modules have dangling
symbol references that are expected to be resolved by whatever dlopen()s
them, but if the perl interpreter itself is loaded via dlopen(RTLD_LOCAL),
the perl symbols are not visible to the very perl extensions that it's
loading, regardless of what flags the perl interpreter itself uses when
calling dlopen().

Stephen and I were just discussing this extensively this morning in
connection with freeradius, which has the same problem in Debian for
rlm_perl; I'm pretty sure this analysis is correct, but I'm open to the
possibility that it works by some means I haven't thought of so am
interested to hear of any test cases you have for this.

BTW, a similar bug was reported recently (... in the past 3 months or so;
and possibly only via the mailing list) on the OpenLDAP package because
back_meta references symbols from back_ldap but doesn't link against it, so
back_meta in Debian can never see those symbols even if back_ldap is loaded
first. I have a possible fix for this (which btw is detected by upstream's
'make test'), but have been reluctant to implement it so far because it's an
ugly thing involving making symlinks to back_ldap to keep libtool happy.

Ultimately, I think it was a poor decision for the libltdl authors to create
an interface that doesn't support specifying crucial flags like RTLD_GLOBAL
and RTLD_LOCAL when opening objects, as this has left us with use cases
which have mutually exclusive requirements. Perhaps it's time to try to
persuade libltdl upstream to add a fixed interface.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu...

Read more...

Revision history for this message
Russ Allbery (rra-debian) wrote :

As mentioned in the associated Debian bug, I believe this problem is caused by the way that Perl modules are linked only on i386. I had closed the Debian bug because it was working on amd64, where Perl modules are all explicitly linked with libperl. On i386, but only i386, they aren't and instead all the Perl symbols are left undefined to be resolved at run-time, which doesn't work inside modules loaded by libtool.

If this doesn't work for anyone on a platform *other* than i386, please let me know, since that means my theory is wrong.

Otherwise, I think the best solution would be to make Perl on i386 match the other architectures. This linking policy was put in place some time back for reasons that appear to now be obscure, and I doubt it's still needed.

Revision history for this message
In , Russ Allbery (rra-debian) wrote :

Steve Langasek <email address hidden> writes:

> Did you test the perl backend using any XS modules?

I did, yes, but only on amd64.

> Since the perl package in Debian is not linked against libperl, perl
> extensions also cannot be linked against libperl since this would cause
> two copies of the perl XS symbols to be loaded.

And this is the problem. It *is* linked against libperl on every
architecture except i386, and on those architectures, the perl binary is
also linked against libperl instead of duplicating the code. I think i386
should be changed to match. This came up on debian-devel recently.

> BTW, a similar bug was reported recently (... in the past 3 months or
> so; and possibly only via the mailing list) on the OpenLDAP package
> because back_meta references symbols from back_ldap but doesn't link
> against it, so back_meta in Debian can never see those symbols even if
> back_ldap is loaded first. I have a possible fix for this (which btw is
> detected by upstream's 'make test'), but have been reluctant to
> implement it so far because it's an ugly thing involving making symlinks
> to back_ldap to keep libtool happy.

Can't we just link back_meta against back_ldap? Or is that what your fix
for the problem does?

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Steve Langasek (vorlon)
Changed in openldap2.3:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Steve Langasek (vorlon) wrote :

Marked as invalid for OpenLDAP upstream; this bug is specific to the Debian/Ubuntu packaging, which uses a system libltdl which specifies RTLD_LOCAL instead of the libltdl upstream default of RTLD_GLOBAL.

Changed in openldap:
status: New → Invalid
Revision history for this message
In , Steve Langasek (vorlon) wrote : Re: Bug#327585: [Pkg-openldap-devel] Bug#327585: reopening

On Sun, Feb 03, 2008 at 05:47:01PM -0800, Russ Allbery wrote:

> > steve@spartacus:~$ sudo /etc/init.d/slapd start
> > Starting OpenLDAP: slapd - failed:
> > Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
> > at /usr/lib/perl/5.8/POSIX.pm line 26
> > Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
> > BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
> > Compilation failed in require at (eval 2) line 1.
> > BEGIN failed--compilation aborted at (eval 2) line 1.

> > Can't call method "config" on an undefined value.

> Okay, thanks.

> I could have sworn that this wasn't happening with 2.3.30 or
> thereabouts... oh, I know what the difference is. I was testing on
> amd64. Right, now I understand.

> This is that problem that's caused by Perl modules not being linked with
> libperl on i386 (but not on any other architecture) and instead
> duplicating the symbols between perl and libperl and leaving them
> undefined in the modules. This works correctly if the modules are linked
> with libperl, which they are on amd64 and Debian's other architectures.

> You're right, this isn't closed; it's just i386-specific.

I was surprised to hear that perl modules were linked to libperl on !i386
(which would be the correct thing to do, I just didn't think the standard
perl build rules handled this). So I checked, and on my amd64 system:

$ ldd /usr/lib/perl/5.8.8/auto/POSIX/POSIX.so
 libm.so.6 => /lib/libm.so.6 (0x00002b02d4451000)
 libc.so.6 => /lib/libc.so.6 (0x00002b02d46d2000)
 /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
$

So sure enough, I tried the test case from
<https://bugs.launchpad.net/ubuntu/+source/openldap2.3/+bug/90812>, and:

# slapd
Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
 at /usr/lib/perl/5.8/POSIX.pm line 26
Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
Compilation failed in require at (eval 2) line 1.
BEGIN failed--compilation aborted at (eval 2) line 1.
#

I have no idea if this is a regression from previous perl builds, but it
looks to me like it's a general problem at this point.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
In , Steve Langasek (vorlon) wrote : Re: Bug#327585: [Pkg-openldap-devel] Bug#327585: Bug#327585: reopening

On Sun, Feb 03, 2008 at 06:02:34PM -0800, Russ Allbery wrote:
> > Since the perl package in Debian is not linked against libperl, perl
> > extensions also cannot be linked against libperl since this would cause
> > two copies of the perl XS symbols to be loaded.

> And this is the problem. It *is* linked against libperl on every
> architecture except i386, and on those architectures, the perl binary is
> also linked against libperl instead of duplicating the code. I think i386
> should be changed to match. This came up on debian-devel recently.

I agree that we're better off if perl links to libperl on all archs, so we
can maximize the use of RTLD_LOCAL. I think libltdl's API should also be
fixed.

> > BTW, a similar bug was reported recently (... in the past 3 months or
> > so; and possibly only via the mailing list) on the OpenLDAP package
> > because back_meta references symbols from back_ldap but doesn't link
> > against it, so back_meta in Debian can never see those symbols even if
> > back_ldap is loaded first. I have a possible fix for this (which btw is
> > detected by upstream's 'make test'), but have been reluctant to
> > implement it so far because it's an ugly thing involving making symlinks
> > to back_ldap to keep libtool happy.

> Can't we just link back_meta against back_ldap? Or is that what your fix
> for the problem does?

Yeah, that's the fix; but libtool --install blows up nastily when you link
against something that's not a system library, *and* has a name that doesn't
begin with "lib", because -lback_ldap doesn't work very well for
relinking... :)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
In , Russ Allbery (rra-debian) wrote : Re: Bug#327585: reopening

Steve Langasek <email address hidden> writes:

> I was surprised to hear that perl modules were linked to libperl on !i386
> (which would be the correct thing to do, I just didn't think the standard
> perl build rules handled this). So I checked, and on my amd64 system:
>
> $ ldd /usr/lib/perl/5.8.8/auto/POSIX/POSIX.so
> libm.so.6 => /lib/libm.so.6 (0x00002b02d4451000)
> libc.so.6 => /lib/libc.so.6 (0x00002b02d46d2000)
> /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
> $
>
> So sure enough, I tried the test case from
> <https://bugs.launchpad.net/ubuntu/+source/openldap2.3/+bug/90812>, and:
>
> # slapd
> Error Can't load '/usr/lib/perl/5.8/auto/POSIX/POSIX.so' for module POSIX: /usr/lib/perl/5.8/auto/POSIX/POSIX.so: undefined symbol: PL_sig_name at /usr/lib/perl/5.8/XSLoader.pm line 70.
> at /usr/lib/perl/5.8/POSIX.pm line 26
> Compilation failed in require at /etc/perl/SampleLDAP.pm line 2.
> BEGIN failed--compilation aborted at /etc/perl/SampleLDAP.pm line 2.
> Compilation failed in require at (eval 2) line 1.
> BEGIN failed--compilation aborted at (eval 2) line 1.
> #
>
> I have no idea if this is a regression from previous perl builds, but it
> looks to me like it's a general problem at this point.

Huh. I feel like I'm losing my mind. I could have sworn that I checked
this at one point and the modules were linked with libperl on amd64, but
maybe I'm dreaming. /usr/bin/perl is liked with libperl on amd64 and not
on i386, but yeah, even in stable the modules aren't.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Changed in openldap2.2:
status: Fix Released → New
Revision history for this message
In , Russ Allbery (rra-debian) wrote : Re: Bug#327585: libltdl and RTLD_GLOBAL

Steve Langasek <email address hidden> writes:
> On Sun, Feb 03, 2008 at 06:02:34PM -0800, Russ Allbery wrote:

>> Can't we just link back_meta against back_ldap? Or is that what your
>> fix for the problem does?

> Yeah, that's the fix; but libtool --install blows up nastily when you
> link against something that's not a system library, *and* has a name
> that doesn't begin with "lib", because -lback_ldap doesn't work very
> well for relinking... :)

I wonder if it would be worthwhile to live with the code duplication and
namespace pollution and use libltdl in convenience mode instead of
installable mode with slapd. That would basically cause slapd to use
RTLD_GLOBAL, correct? It would resolve the issue with back-perl and also
with back-meta, at the cost of namespace pollution only for slapd plugins
and things they load.

Of course, that means that if someone tries to use a Perl module that uses
unixodbc via back-perl, we probably lose again for all the same reasons
that Debian moved away from RTLD_GLOBAL in the first place, but it might
be a workaround.

I'm not sure how hard it would be to fix the Perl linking problem, and the
linking issues with back-meta sound pretty annoying.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Revision history for this message
In , Russ Allbery (rra-debian) wrote : Re: [Pkg-openldap-devel] Bug#327585: libltdl and RTLD_GLOBAL

Russ Allbery <email address hidden> writes:

> I wonder if it would be worthwhile to live with the code duplication and
> namespace pollution and use libltdl in convenience mode instead of
> installable mode with slapd. That would basically cause slapd to use
> RTLD_GLOBAL, correct? It would resolve the issue with back-perl and
> also with back-meta, at the cost of namespace pollution only for slapd
> plugins and things they load.

For back-meta, never mind -- Howard Chu says that back-meta is violating
the module policy for slapd and they will want to change it upstream so
that it doesn't have the direct dependency on back-ldap.

That still leaves back-perl. Maybe we can convince Brendan to start
linking all modules (and the Perl interpretor) against libperl as of the
5.10 release that's coming soon?

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Revision history for this message
In , Stephen Gran (sgran) wrote :

This one time, at band camp, Russ Allbery said:
> Russ Allbery <email address hidden> writes:
>
> > I wonder if it would be worthwhile to live with the code duplication and
> > namespace pollution and use libltdl in convenience mode instead of
> > installable mode with slapd. That would basically cause slapd to use
> > RTLD_GLOBAL, correct? It would resolve the issue with back-perl and
> > also with back-meta, at the cost of namespace pollution only for slapd
> > plugins and things they load.
>
> For back-meta, never mind -- Howard Chu says that back-meta is violating
> the module policy for slapd and they will want to change it upstream so
> that it doesn't have the direct dependency on back-ldap.
>
> That still leaves back-perl. Maybe we can convince Brendan to start
> linking all modules (and the Perl interpretor) against libperl as of the
> 5.10 release that's coming soon?

I honestly don't know if it makes sense or not. When this was brought
up, several people advanced several theories about why it might be done
this way, and there were as many theories as people. I'd be curious in
the general case why perl is built that way.

If it turns out that it can be built so that the binary is linked to
libperl on all arches, linking the modules is fairly straightforward -
changing MakeMaker to always link to -lperl should do it for all the
binary modules. We'll need to trigger a massive binNMU afterwards, but
that should be doable.

Do you feel like getting the ball rolling with Bod ?

Cheers,
--
 -----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : <email address hidden> |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
 -----------------------------------------------------------------

Revision history for this message
In , Steve Langasek (vorlon) wrote : Re: Bug#327585: [Pkg-openldap-devel] Bug#327585: libltdl and RTLD_GLOBAL

On Thu, Feb 07, 2008 at 12:02:27PM +0000, Stephen Gran wrote:

> > That still leaves back-perl. Maybe we can convince Brendan to start
> > linking all modules (and the Perl interpretor) against libperl as of the
> > 5.10 release that's coming soon?

> I honestly don't know if it makes sense or not. When this was brought
> up, several people advanced several theories about why it might be done
> this way, and there were as many theories as people. I'd be curious in
> the general case why perl is built that way.

Because libperl is built as PIC code, which would be substantial overhead
for the perl interpreter on i386.

In my opinion, this shouldn't be a sufficient rationale to block getting
perl DSOs linked right, but then I have rather strong opinions about DSOs
and have to regard myself as biased. :)

> If it turns out that it can be built so that the binary is linked to
> libperl on all arches, linking the modules is fairly straightforward -
> changing MakeMaker to always link to -lperl should do it for all the
> binary modules. We'll need to trigger a massive binNMU afterwards, but
> that should be doable.

Even if i386 doesn't start linking to libperl, it seems to me that the
MakeMaker change should be doable on a per-arch basis (though preferably
without having to move perl-modules from Arch: all to Arch: any).

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
In , Russ Allbery (rra-debian) wrote : retitle 327585 to perl backend failed to load XS due to library symbol scope

# Automatically generated email from bts, devscripts version 2.10.13
retitle 327585 perl backend failed to load XS due to library symbol scope

Revision history for this message
In , Steve Langasek (vorlon) wrote : Re: Bug#327585: libltdl and RTLD_GLOBAL

On Wed, Feb 06, 2008 at 09:18:13PM -0800, Russ Allbery wrote:
> Steve Langasek <email address hidden> writes:
> > On Sun, Feb 03, 2008 at 06:02:34PM -0800, Russ Allbery wrote:

> >> Can't we just link back_meta against back_ldap? Or is that what your
> >> fix for the problem does?

> > Yeah, that's the fix; but libtool --install blows up nastily when you
> > link against something that's not a system library, *and* has a name
> > that doesn't begin with "lib", because -lback_ldap doesn't work very
> > well for relinking... :)

> I wonder if it would be worthwhile to live with the code duplication and
> namespace pollution and use libltdl in convenience mode instead of
> installable mode with slapd. That would basically cause slapd to use
> RTLD_GLOBAL, correct? It would resolve the issue with back-perl and also
> with back-meta, at the cost of namespace pollution only for slapd plugins
> and things they load.

> Of course, that means that if someone tries to use a Perl module that uses
> unixodbc via back-perl, we probably lose again for all the same reasons
> that Debian moved away from RTLD_GLOBAL in the first place, but it might
> be a workaround.

Hmm, now that we're build-depending on UnixODBC instead of iODBC, it might
also cause problems for back_sql. If you think this is the right solution
otherwise, we could revert back to iodbc for the time being. (I seem to
have a bug to shake out there anyway, openldap2.3 as it stands seems to be
missing a build-conflicts with libiodbc2...)

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Revision history for this message
In , Russ Allbery (rra-debian) wrote : Re: [Pkg-openldap-devel] Bug#327585: libltdl and RTLD_GLOBAL

Steve Langasek <email address hidden> writes:

> Hmm, now that we're build-depending on UnixODBC instead of iODBC, it
> might also cause problems for back_sql. If you think this is the right
> solution otherwise, we could revert back to iodbc for the time being.
> (I seem to have a bug to shake out there anyway, openldap2.3 as it
> stands seems to be missing a build-conflicts with libiodbc2...)

I think the right solution is to change Perl, honestly. I just don't have
any time to pursue that with Brendan.

--
Russ Allbery (<email address hidden>) <http://www.eyrie.org/~eagle/>

Revision history for this message
In , Steve Langasek (vorlon) wrote : tagging 327585

# Automatically generated email from bts, devscripts version 2.10.33
tags 327585 confirmed

Changed in openldap2.2:
status: New → Confirmed
Mathias Gug (mathiaz)
Changed in openldap:
importance: Undecided → Medium
status: New → Triaged
Steve Langasek (vorlon)
Changed in openldap2.3:
status: Triaged → Invalid
Changed in openldap2.2:
status: Triaged → Invalid
Revision history for this message
In , Boris Lechner (boris-lechner) wrote : Is there any workaround ?

Hello,

as I met this bug too, I'd like to know if someone found a workaround ?

---------------------------------------------------
 Boris LECHNER
 Administrateur systeme de la Plateforme Mecanique
 Institut National des Sciences Appliquées
 24 bvd de la Victoire 67084 Strasbourg Cedex
 Tel : 03 88 14 49 59
 Fax : 03 88 14 47 99
---------------------------------------------------

Revision history for this message
Davor Ocelic (docelic) wrote :

Just a note for those wanting to run slapd perl backed right away-- the
solution is simply to preload libperl in the slapd startup command like this:

LD_PRELOAD=/usr/lib/libperl.so.5.10 /usr/sbin/slapd -d 256.....

Revision history for this message
In , Niko Tyni (ntyni) wrote : Re: embedding perl, libltdl and RTLD_GLOBAL
Download full text (4.1 KiB)

Hi,

I've been looking at the "libltdl and RTLD_GLOBAL" issue with embedding
perl in a dlopen'd plugin.

An instance of this with freeradius is #416266 (recently reassigned to
perl), and I see #327585 against openldap is another one.

To recap, the problem is that lt_dlopen() from the Debian system libltdl
has called dlopen(3) with RTLD_LOCAL instead of RTLD_GLOBAL ever since
#195821 was fixed. As the compiled XS modules aren't linked against
libperl, its symbols aren't exposed to them, resulting in errors like
'/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp'.

Observations:

- this problem isn't specific to perl and can easily be triggered with the
  freeradius rlm_python module too [1]

- it's clearly possible to dlopen() compiled Perl modules from a dlopen'd
  module if you don't use libltdl, see apache2+libapache2-mod-perl2 for
  an example

- the XS modules are actually plugins in a private directory, not generic
  shared libraries. Having unresolved symbols in a plugin without
  a corresponding NEEDED entry seems to be very common, see for
  example /usr/lib/apache2/modules, /usr/lib/python2.5/lib-dynload/,
  /usr/lib/cdebconf etc.

- as noted in #327585, linking the XS shared objects against libperl
  is potentially a problem on *i386, where /usr/bin/perl is statically
  linked with libperl.a for performance reasons. (I don't have any data
  about these performance reasons myself, I'm relying on hearsay and
  /usr/share/doc/perl/README.Debian.gz here.)

  While this does seem to work in a quick and limited test of mine,
  it would bring in both libperl.a and libperl.so for all uses of
  /usr/bin/perl that need XS modules, and I'm not sure which version of
  the functions would get used later. If the PIC versions win, we'd be
  giving away the performance benefit we got from static linking in the
  first place.

  At the very least, it would add 1.5M to the size of the perl-base
  package on i386 AFAICS. I'm not sure how much the memory footprint of
  the /usr/bin/perl invocations would increase.

  Also note that we currently ship /usr/lib/libperl.a on all the
  architectures, so everything that applies to the i386 /usr/bin/perl
  case applies to anybody using the static library on the other archs too.

  Given that i386 is still our most popular architecture, the other
  proposed options don't seem very appealing either:
    * only link the modules against libperl.so on the other architectures
      (no fix for i386)
    * link /usr/bin/perl dynamically on i386 too
      (reduced performance in the very common case
       for the benefit of a very uncommon case)

- it turns out libltdl nowadays does have an interface where you can
  specify RTLD_GLOBAL. From the libtool Changelog.2007:

    2007-05-08 Gary V. Vaughan <email address hidden>

        Without this patch, lt_dlopen always opens modules with symbol
        visibility set according to the underlying implementation.
        Here, we add lt_dlopenadvise() to allow callers to request,
        among other things, local or global symbol visibility from the
        underlying dlloader:

  Indeed, the attached proof of concept makes the freeradius problem g...

Read more...

Revision history for this message
In , Josip Rodin (joy-debbugs) wrote :
Download full text (6.0 KiB)

Alan, what do you think about the below? The *advise example is right out
of libtool documentation :)

I've noticed this local commit:
http://github.com/alandekok/freeradius-server/commit/4df74f9b1497fc4c88f9159a680707041c70a23d

Maybe it's about a similar issue?

On Mon, Nov 30, 2009 at 12:08:20PM +0200, Niko Tyni wrote:
> I've been looking at the "libltdl and RTLD_GLOBAL" issue with embedding
> perl in a dlopen'd plugin.
>
> An instance of this with freeradius is #416266 (recently reassigned to
> perl), and I see #327585 against openldap is another one.
>
> To recap, the problem is that lt_dlopen() from the Debian system libltdl
> has called dlopen(3) with RTLD_LOCAL instead of RTLD_GLOBAL ever since
> #195821 was fixed. As the compiled XS modules aren't linked against
> libperl, its symbols aren't exposed to them, resulting in errors like
> '/usr/lib/perl/5.10/auto/Data/Dumper/Dumper.so: undefined symbol: Perl_sv_cmp'.
>
> Observations:
>
> - this problem isn't specific to perl and can easily be triggered with the
> freeradius rlm_python module too [1]
>
> - it's clearly possible to dlopen() compiled Perl modules from a dlopen'd
> module if you don't use libltdl, see apache2+libapache2-mod-perl2 for
> an example
>
> - the XS modules are actually plugins in a private directory, not generic
> shared libraries. Having unresolved symbols in a plugin without
> a corresponding NEEDED entry seems to be very common, see for
> example /usr/lib/apache2/modules, /usr/lib/python2.5/lib-dynload/,
> /usr/lib/cdebconf etc.
>
> - as noted in #327585, linking the XS shared objects against libperl
> is potentially a problem on *i386, where /usr/bin/perl is statically
> linked with libperl.a for performance reasons. (I don't have any data
> about these performance reasons myself, I'm relying on hearsay and
> /usr/share/doc/perl/README.Debian.gz here.)
>
> While this does seem to work in a quick and limited test of mine,
> it would bring in both libperl.a and libperl.so for all uses of
> /usr/bin/perl that need XS modules, and I'm not sure which version of
> the functions would get used later. If the PIC versions win, we'd be
> giving away the performance benefit we got from static linking in the
> first place.
>
> At the very least, it would add 1.5M to the size of the perl-base
> package on i386 AFAICS. I'm not sure how much the memory footprint of
> the /usr/bin/perl invocations would increase.
>
> Also note that we currently ship /usr/lib/libperl.a on all the
> architectures, so everything that applies to the i386 /usr/bin/perl
> case applies to anybody using the static library on the other archs too.
>
> Given that i386 is still our most popular architecture, the other
> proposed options don't seem very appealing either:
> * only link the modules against libperl.so on the other architectures
> (no fix for i386)
> * link /usr/bin/perl dynamically on i386 too
> (reduced performance in the very common case
> for the benefit of a very uncommon case)
>
> - it turns out libltdl nowadays does have an interface where you can
> specify RTLD_GLOBAL. From the libtool Changel...

Read more...

Revision history for this message
In , Alan DeKok (aland-deployingradius) wrote :

On 09-12-09 12:23 PM, Josip Rodin wrote:
> Alan, what do you think about the below? The *advise example is right out
> of libtool documentation :)

  It looks OK, *if* libltdl has the lt_dladvise() functions.

> I've noticed this local commit:
> http://github.com/alandekok/freeradius-server/commit
/4df74f9b1497fc4c88f9159a680707041c70a23d
>
> Maybe it's about a similar issue?

  Nope. That commit was to fix an issue where libltdl would crash.
i.e. not return "failed linking to X", but *die*, and take the
application down with it.

  Alan DeKok.

Revision history for this message
In , Josip Rodin (joy-debbugs) wrote :

On Wed, Dec 09, 2009 at 12:34:10PM +0100, Alan DeKok wrote:
> On 09-12-09 12:23 PM, Josip Rodin wrote:
> > Alan, what do you think about the below? The *advise example is right out
> > of libtool documentation :)
>
> It looks OK, *if* libltdl has the lt_dladvise() functions.

Aha. The libtool NEWS file says it first appeared in version 2.1b
(2008-02-01), and the ChangeLog file says the patch has been in libtool VCS
since 2007-05-08. So it's not particularly novel, but we would need to up
the dependencies a bit. You could update the internal copy of libltdl in the
non-bleeding-edge branch, too. Combined with those other changes I
previously suggested, that could work.

--
     2. That which causes joy or happiness.

Revision history for this message
In , Alan DeKok (aland-deployingradius) wrote :

On 09-12-09 1:06 PM, Josip Rodin wrote:
> Aha. The libtool NEWS file says it first appeared in version 2.1b
> (2008-02-01), and the ChangeLog file says the patch has been in libtool VCS
> since 2007-05-08. So it's not particularly novel, but we would need to up
> the dependencies a bit. You could update the internal copy of libltdl in the
> non-bleeding-edge branch, too. Combined with those other changes I
> previously suggested, that could work.

  I've committed a fix to 2.1.8. It requires a new macro to be defined:
HAVE_LT_DLADVISE_INIT. Once that's defined, it should use the new
lt_dladvise() API.

  I haven't added a "configure" check, because we're pretty close to a
release of 2.1.8. Likewise, I haven't upgraded the internal copy of
libltdl.

  I'll take a look at updating the internal copy of libltdl for 2.2.0,
in a month or two.

  Alan DeKok.

Changed in openldap (Debian):
status: Confirmed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote :

This was fixed in version 2.4.25-4 of the openldap package in Debian, so in Ubuntu is fixed for 12.04 and later.

Changed in openldap (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.