ps-toku-admin says jemalloc is in use but plugin installation fails

[Swany]

[justin@localhost grub2]$ sudo ps_tokudb_admin --enable -u root
Checking if Percona server is running with jemalloc enabled...
>> Percona server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
>> Transparent huge pages are enabled (should be disabled).

Checking if thp-setting=never option is already set in config file...
>> Option thp-setting=never is not set in the config file.
>> (needed only if THP is not disabled permanently on the system)

Checking TokuDB plugin status...
>> TokuDB plugin is not installed.

Disabling transparent huge pages for the current session...
>> Successfuly disabled transparent huge pages for this session.

Adding thp-setting=never option into /etc/my.cnf
>> Successfuly added thp-setting=never option into /etc/my.cnf

Installing TokuDB engine...
>> Error installing TokuDB plugin. Please check error log.

[justin@localhost grub2]$ sudo cat /var/log/mysqld.log
150216 19:49:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2015-02-16 19:49:13 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-02-16 19:49:13 2133 [Note] Plugin 'FEDERATED' is disabled.
2015-02-16 19:49:13 2133 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-02-16 19:49:13 2133 [Note] InnoDB: The InnoDB memory heap is disabled
2015-02-16 19:49:13 2133 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-02-16 19:49:13 2133 [Note] InnoDB: Memory barrier is not used
2015-02-16 19:49:13 2133 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-02-16 19:49:13 2133 [Note] InnoDB: Using Linux native AIO
2015-02-16 19:49:13 2133 [Note] InnoDB: Not using CPU crc32 instructions
2015-02-16 19:49:13 2133 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-02-16 19:49:13 2133 [Note] InnoDB: Completed initialization of buffer pool
2015-02-16 19:49:13 2133 [Note] InnoDB: Highest supported file format is Barracuda.
2015-02-16 19:49:13 2133 [Note] InnoDB: 128 rollback segment(s) are active.
2015-02-16 19:49:13 2133 [Note] InnoDB: Waiting for purge to start
2015-02-16 19:49:13 2133 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 1625987
2015-02-16 19:49:13 2133 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: cacde206-b63e-11e4-ac95-d443d04b063c.
2015-02-16 19:49:13 2133 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2015-02-16 19:49:13 2133 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2015-02-16 19:49:13 2133 [Note] Server hostname (bind-address): '*'; port: 3306
2015-02-16 19:49:13 2133 [Note] IPv6 is available.
2015-02-16 19:49:13 2133 [Note] - '::' resolves to '::';
2015-02-16 19:49:13 2133 [Note] Server socket created on IP: '::'.
2015-02-16 19:49:13 2133 [Note] Event Scheduler: Loaded 0 events
2015-02-16 19:49:13 2133 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.22-72.0' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 72.0, Revision 738
2015-02-16 20:04:47 2133 [ERROR] TokuDB is not initialized because jemalloc is not loaded
2015-02-16 20:04:47 2133 [ERROR] Plugin 'TokuDB' init function returned error.
2015-02-16 20:04:47 2133 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
2015-02-16 20:04:47 2133 [Note] Shutting down plugin 'TokuDB'

All I did was:
sudo yum install Percona-Server-*56*
Then ps-toku-admin --enable -u root

Expected result:
working tokudb after basic RPM setup

[Swany (greenlion)]

So this is caused by SElinux. It won't allow the LD_PRELOAD to work. The tool should error out if SElinux is enabled or it should modify the SElinux configuration to allow the database to work.

[Valerii Kravchuk (valerii-kravchuk)]

Exact RHEL/CentOS version is missing here. On CentOS 6.6, for example, everything works as expected. The problem probably happens on RHEL 7.

[Valerii Kravchuk (valerii-kravchuk)]

[root@centos openxs]# sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: permissive
Policy version: 24
Policy from config file: targeted

But I still do not see AVC messages related to LD_PRELOAD or mysqld or anything on this CentOS 6.6.

[Swany (greenlion)]

I'm using CentOS 7. Selinux defaults to enforcing, not permissive.

[Swany (greenlion)]

[justin@localhost ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: error (Success)
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

sudo cat /var/log/mysqld.log|grep jemalloc
2015-02-19 08:40:43 4262 [ERROR] TokuDB is not initialized because jemalloc is not loaded

[justin@localhost ~]$ sudo setenforce 0
[justin@localhost ~]$ sudo service mysqld stop
Redirecting to /bin/systemctl stop mysqld.service
[justin@localhost ~]$ sudo rm /var/log/mysqld.log
[justin@localhost ~]$ sudo service mysqld start
Redirecting to /bin/systemctl start mysqld.service
[justin@localhost ~]$ sudo cat /var/log/mysqld.log |grep jemalloc
<no matches>

[David Bennett (dbpercona)]

mysqld_safe must be allowed to modify the process environment with:

module mysqld_safe_preload 0.1;
require {
  type mysqld_safe_t;
  type mysqld_t;
  class process { noatsecure } ;
}
allow mysqld_safe_t mysqld_t:process { noatsecure };

[Nilnandan Joshi (nilnandan-joshi)]

Verified with CentOS 7. If we'll set from enforcing to permissive, we'll be able to install it.

[root@localhost ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# sudo ps_tokudb_admin --enable -u root
Checking if Percona server is running with jemalloc enabled...
>> Error checking pid file location! Please check username, password and other options...
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# ps_tokudb_admin --enable -uroot -proot
Checking if Percona server is running with jemalloc enabled...
>> Percona server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
>> Transparent huge pages are enabled (should be disabled).

Checking if thp-setting=never option is already set in config file...
>> Option thp-setting=never is not set in the config file.
>> (needed only if THP is not disabled permanently on the system)

Checking TokuDB plugin status...
>> TokuDB plugin is not installed.

Disabling transparent huge pages for the current session...
>> Successfuly disabled transparent huge pages for this session.

Adding thp-setting=never option into /etc/my.cnf
>> Successfuly added thp-setting=never option into /etc/my.cnf

Installing TokuDB engine...
>> Error installing TokuDB plugin. Please check error log.

[root@localhost ~]#
[root@localhost ~]# tail -15 /var/lib/mysql/mysqld.log
2015-02-24 17:18:53 2041 [Note] InnoDB: Waiting for purge to start
2015-02-24 17:18:53 2041 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 1626077
2015-02-24 17:18:53 2041 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2015-02-24 17:18:53 2041 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2015-02-24 17:18:53 2041 [Note] Server hostname (bind-address): '*'; port: 3306
2015-02-24 17:18:53 2041 [Note] IPv6 is available.
2015-02-24 17:18:53 2041 [Note] - '::' resolves to '::';
2015-02-24 17:18:53 2041 [Note] Server socket created on IP: '::'.
2015-02-24 17:18:54 2041 [Note] Event Scheduler: Loaded 0 events
2015-02-24 17:18:54 2041 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.22-72.0' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 72.0, Revision 738
2015-02-24 17:20:44 2041 [ERROR] TokuDB is not initialized because jemalloc is not loaded
2015-02-24 17:20:44 2041 [ERROR] Plugin 'TokuDB' init function returned error.
2015-02-24 17:20:44 2041 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
2015-02-24 17:20:44 2041 [Note] Shutting down plugin 'TokuDB'
[root@localhost ~]#

[Roel Van de Paar (roel11)]

If the script is able to detect PS is running with jemalloc (>> Percona server is running with jemalloc enabled.) then somehow it seems that TokuDB init should be able to use this same info instead of failing (/instead of having to use LD_PRELOAD)... or?

[Swany (greenlion)]

I think it detects it is configured to be automatically ld_preloaded by mysqld_safe, not that it is actually being used by server. Ld_preload fails because of selinux.

Sent from my iPhone

> On Mar 15, 2015, at 7:04 PM, Roel Van de Paar <email address hidden> wrote:
>
> If the script is able to detect PS is running with jemalloc (>> Percona
> server is running with jemalloc enabled.) then somehow it seems that
> TokuDB init should be able to use this same info instead of failing
> (/instead of having to use LD_PRELOAD)... or?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1422538
>
> Title:
> ps-toku-admin says jemalloc is in use but plugin installation fails
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-server/+bug/1422538/+subscriptions

[David Bennett (dbpercona)]

The detection looks at the environment of mysqld to see if jemalloc was loaded via LD_PRELOAD:

  grep -c jemalloc /proc/${PID_NUM}/environ

The problem with SELinux is that even though jemalloc is in the LD_PRELOAD, SELinux is preventing the library from hooking the allocation functions as part of it's secure mode environment cleansing (AT_SECURE). When the TokuDB storage engine plug-in detects jemalloc from inside the mysqld process space, it's not there.

So the real problem here is the SELinux policy included in the RHEL/Centos PS packaging.

 The noatsecure policy in comment #6 will fix this issue.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Might as well be good to add selinux detection to the ps-toku-admin script and warn if no selinux policies are present (in future).

[Roel Van de Paar (roel11)]

This bug prevents TokuDB, ps_tokudb_admin and TokuBackup from working on Vanilla Centos 7.

[Tomislav Plavcic (tplavcic)]

This is fixed currently so that the script warns about the selinux in enforcing state. Once the selinux policies are published it should be changed and tested more.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-872