Binary build is linked against libssl.so.10 and libcrypto.so.10 making it hard to run on e.g. SLES11

[Al T]

I've posted a comment here https://bugs.launchpad.net/percona-server/+bug/1172916/comments/14 on a similar bug for Percona-Server, but as long as PXC was explicitly removed from there and the bug has "fix released" state, I guess it was not the best place to do that.

The point is that some of the server distributions, I saw that on latest SP of SLES11, still have openssl 0.9.8 and therefore don't have these two libraries. I've successfully created symlinks with the above names to corresponding 0.9.8 libraries and I see that both MySQL SSL and Galera SSL are working fine, meaning there's no real need for 1.0 version. But using such a trick is worrisome for production workloads, so I would propose just to link PXC binary distribution against 0.9.8.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Al,

Has the workaround of " yum install openssl098e" not worked in SLES? This bug, if fixed in PS, will be in next PXC release.

[Al T (al-t)]

That unfortunately doesn't work for SLES, (there's no yum there at all, but never mind, I get the idea). I do have the openssl libraries installed (it's called libopenssl0_9_8, main openssl package doesn't provide the libopenssl.so.* files in SLES), but it's just the case that the package only contains libcrypto.so.0.9.8 and libssl.so.0.9.8, while PXC wants to see libcrypto.so.10 and libssl.so.10 respectively.

Glad to see the fix is going to be included into PXC 5.5.33, I'll definitely test it out and report back when it's available. Thanks for your help.

[Raghavendra D Prabhu (raghavendra-prabhu)]

It has been fixed in Percona Server in https://bazaar.launchpad.net/~percona-core/percona-server/release-5.5.33-31.1/revision/565

Building PXC statically with SSL for binaries should do.

[Ignacio Nin (ignacio-nin)]

I've created static-openssl jobs for PXC, as there are for PS, these partially statically linked binaries should fix this issue.

PoC binaries can be found at http://www.percona.com/downloads/TESTING/Percona-XtraDB-Cluster/5.5.33-23.7.5/release-5.5.33/476/binary/linux/.

[Al T (al-t)]

Yes, looks like this one fixes the problem - thanks Ignacio.

[Al T (al-t)]

Looks like it's not completely fixed, unfortunately.

The MySQL binaries themselves are now ok, but the bundled Galera library (libgalera_smm.so) is still dynamically linked to libssl.so.10 and libcrypto.so.10, thus it doesn't start:

ldd /mysql/pxc_5.5.33_3306/lib/libgalera_smm.so
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaaecb000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaab0e8000)
        libssl.so.10 => not found
        libcrypto.so.10 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aaaab2f2000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaab5fd000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab876000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaaba8c000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

Log contents:
131009 13:49:13 mysqld_safe Starting mysqld daemon with databases from /mysql/data
131009 13:49:13 mysqld_safe Skipping wsrep-recover for empty datadir: /mysql/data
131009 13:49:13 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
131009 13:49:13 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
131009 13:49:13 [Note] WSREP: Read nil XID from storage engines, skipping position init
131009 13:49:13 [Note] WSREP: wsrep_load(): loading provider library '/mysql/lib/libgalera_smm.so'
131009 13:49:13 [ERROR] WSREP: wsrep_load(): dlopen(): libssl.so.10: cannot open shared object file: No such file or directory
131009 13:49:13 [ERROR] WSREP: wsrep_load(/mysql/lib/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
131009 13:49:13 [Note] WSREP: Read nil XID from storage engines, skipping position init
131009 13:49:13 [Note] WSREP: wsrep_load(): loading provider library 'none'
131009 13:49:13 [ERROR] Aborting

131009 13:49:13 [Note] WSREP: Service disconnected.
131009 13:49:14 [Note] WSREP: Some threads may fail to exit.
131009 13:49:14 [Note] /mysql/bin/mysqld: Shutdown complete

[Al T (al-t)]

Sorry, ldd path is incorrect, should be

ldd /mysql/lib/libgalera_smm.so
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaaecb000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaab0e8000)
        libssl.so.10 => not found
        libcrypto.so.10 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aaaab2f2000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaab5fd000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab876000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaaba8c000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

[Al T (al-t)]

Moreover, the trick with creating symlinks (e.g. libssl.so.10 -> /usr/lib64/libssl.so.0.9.8) which worked with PXC 5.5.31 now doesn't work. I understand that was an ugly workaround anyway, but at least it worked quite well (all SSL features worked ok), now it's unusable.

 MySQL crashes with the following in the log:

<...>
131009 14:25:24 [Note] WSREP: Passing config to GCS: base_host = 192.168.56.101; base_port = 4567; cert.log_conflicts = no; gcache.dir = /mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gc
s.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
13:25:24 UTC - mysqld got signal 11 ;
<...>
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x80e0d5]
/mysql/bin/mysqld(handle_fatal_signal+0x4b4)[0x6e64d4]
/lib64/libpthread.so.0(+0xf7c0)[0x2aaaaacdb7c0]
/mysql/bin/mysqld[0xa1c7e8]
/mysql/bin/mysqld(ssl_create_cipher_list+0x83e)[0xa1dcbe]
/mysql/bin/mysqld(SSL_CTX_new+0x1ca)[0xa17eba]
/mysql/lib/libgalera_smm.so(_ZN4asio3ssl13basic_contextINS0_15context_serviceEEC2ERNS_10io_serviceENS0_12context_base6methodE+0x88)[0x2aaaae903d68]
/mysql/lib/libgalera_smm.so(_ZN6galera3ist8ReceiverC2ERN2gu6ConfigEPKc+0x12f)[0x2aaaae8fbd3f]
/mysql/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMMC1EPK15wsrep_init_args+0xa4d)[0x2aaaae91decd]
/mysql/lib/libgalera_smm.so(galera_init+0x34)[0x2aaaae92f6c4]
/mysql/bin/mysqld(_Z10wsrep_initv+0x6c0)[0x69d890]
/mysql/bin/mysqld(_Z18wsrep_init_startupb+0x10)[0x69db50]
/mysql/bin/mysqld[0x559863]
/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x851)[0x55ccc1]
/lib64/libc.so.6(__libc_start_main+0xe6)[0x2aaaab9ccc16]
/mysql/bin/mysqld[0x54fa0d]

[Al T (al-t)]

And finally - ironically enough, but the fully-dymanically linked tarball works ok with the symlink workaround. Probably the statically compiled in libssl 1.0 in mysqld and fake 1.0 picked up by libgalera don't really work well together.

If you hit the same problem - feel free to use that workaround.

All in all - I'd really appreciate if you could update the statically linked version to have libgalera linked statically too.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Al T,

Thanks for the details. I will open a separate issue for this, however, static linking of galera libraries may not be as simple. As a quicker workaround, I suggest to checkout from bzr+ssh://bazaar.launchpad.net/~percona-dev/percona-xtradb-cluster/galera-2.x/ and build the libgalera_smm.so (that is the only file you will need if you are not using garbd).

and then to build, you can do

scons --config=force boost_pool=0 -j8 libgalera_smm.so.

(boost_pool=0 is optional, use it only if build fails).

Alternatively, you can try with galera RPMs of centos 5, centos5 I believe is still on older libssl, hence should be compatible.

[Al T (al-t)]

@Raghavendra,

Thanks for the hint. I think I'd stay with the symlink workaround and fully dynamically linked PXC build for now - I've tested it rather extensively by now and all SSL functions seem to work + I haven't went all-production with PXC yet, so I'll just wait a bit more with that.

I'm trying to avoid building from sources on my own to save the time and avoid any problems caused by those inevitable tiny differences (and potentially-bugs) in my build chain vs. the one you use.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Al-t,

PXC 5.5.34 binaries are built with openssl of centos 5 (so are galera ones), so the issue with openssl shouldn't be there.

[Al T (al-t)]

Thanks Raghavendra, I'll check it out and let you know

[Al T (al-t)]

I've got to test it out eventually and it looks a little strange, to be honest and apparently not fixed, please see the below ldd runs, they speak for themselves. In short - the "static" tarball is only "static" for Galera, MySQL binaries are still dynamically linked to *.so.10 files and the "dynamic" tarball is dynamically linked to a different version (*.so.6 instead of *.so.10 initially), which also doesn't exist on SLES.

linux:/mysql # ldd pxc_5.5.34_3306/bin/mysql
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaaacd000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaaacea000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaaaf64000)
        libssl.so.6 => not found
        libcrypto.so.6 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab16e000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00002aaaab372000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaab5ba000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
linux:/mysql # ldd pxc_5.5.34_3306/lib/libgalera_smm.so
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaaedb000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaab0f8000)
        libssl.so.6 => not found
        libcrypto.so.6 => not found
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aaaab302000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaab60d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaab886000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaaba9c000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
linux:/mysql # ldd pxc_static_5.5.34_3306/bin/mysql
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaaaacd000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaaacea000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaaaf64000)
        libssl.so.10 => not found
        libcrypto.so.10 => not found
        libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab16e000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00002aaaab372000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaab5ba000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
        libtinfo.so.5 => not found
linux:/mysql # ldd pxc_static_5.5.34_3306/lib/libgalera_smm.so
        linux-vdso.so.1 => (0x00002aaaaaaab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaab0b9000)
        librt.so.1 => /lib64/librt.so.1 (0x00002aaaab2d6000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aaaab4e0000)
        libm.so.6 => /lib64/libm.so.6 (0x00002aaaab7ea000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aaaaba63000)
        libc.so.6 => /lib64/libc.so.6 (0x00002aaaabc7a000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1079