galera server systemd warning: the role sets a limit too low for the amount of open files

[Jean-Philippe Evrard]

Trying to restart my galera cluser on master with ubuntu 16.04, I've got this issue:

root@aio1-galera-container-4f9ab445:/etc/systemd/system# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─limits.conf, migrated-from-my.cnf-settings.conf, timeout.conf
   Active: active (running) since Tue 2017-01-24 16:59:58 UTC; 31min ago
  Process: 9018 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 9016 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
  Process: 8741 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (cod
  Process: 8739 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 8738 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 8976 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─8976 /usr/sbin/mysqld --wsrep_start_position=1cced90c-e243-11e6-9f09-abc8dcff6412:10568

Jan 24 16:59:52 aio1-galera-container-4f9ab445 systemd[1]: Starting MariaDB database server...
Jan 24 16:59:52 aio1-galera-container-4f9ab445 mysqld[8741]: 2017-01-24 16:59:52 140355074636032 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB-1~xenial) starting as process 8944 ...
Jan 24 16:59:56 aio1-galera-container-4f9ab445 mysqld[8741]: WSREP: Recovered position 1cced90c-e243-11e6-9f09-abc8dcff6412:10568
Jan 24 16:59:56 aio1-galera-container-4f9ab445 mysqld[8976]: 2017-01-24 16:59:56 140248216168704 [Note] /usr/sbin/mysqld (mysqld 10.1.21-MariaDB-1~xenial) starting as process 8976 ...
Jan 24 16:59:56 aio1-galera-container-4f9ab445 mysqld[8976]: 2017-01-24 16:59:56 140248216168704 [Warning] Could not increase number of max_open_files to more than 800 (request: 21291)
Jan 24 16:59:58 aio1-galera-container-4f9ab445 systemd[1]: Started MariaDB database server.

As you can see I got max open_files set to 800.

This was computed in our galera_server role and set within the limits.conf file.
This isn't the intended behavior.

We should rely on the templated mysql conf values, and do not set a calculated limit there.
Default package open file limit is 16384.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/424775

[Major Hayden (rackerhacker)]

Thanks for opening this bug -- it was affecting me on CentOS as well.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424775
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=79b39d585ac1941c296dcab930047ceb0da8fd52
Submitter: Jenkins
Branch: master

commit 79b39d585ac1941c296dcab930047ceb0da8fd52
Author: Jean-Philippe Evrard <email address hidden>
Date: Tue Jan 24 17:45:23 2017 +0000

    Set the proper amount of open files in systemd

    We used the amount of connections instead of the maximum number
    of open files for the systemd unit limit.

    The amount of connections are handled by mysql.

    Change-Id: Ib82cba87038ac2601b4ff268c6c889dbe2387228
    Closes-Bug: #1659060

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/425079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425079
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=35310819c95fc0739afd4794c9861d256ae9c85f
Submitter: Jenkins
Branch: stable/newton

commit 35310819c95fc0739afd4794c9861d256ae9c85f
Author: Jean-Philippe Evrard <email address hidden>
Date: Tue Jan 24 17:45:23 2017 +0000

    Set the proper amount of open files in systemd

    We used the amount of connections instead of the maximum number
    of open files for the systemd unit limit.

    The amount of connections are handled by mysql.

    Change-Id: Ib82cba87038ac2601b4ff268c6c889dbe2387228
    Closes-Bug: #1659060
    (cherry picked from commit 79b39d585ac1941c296dcab930047ceb0da8fd52)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 15.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-galera_server 14.0.8 release.