mysql-systemd-start causes issues with install and MySQL initialisation

[Ceri WIlliams]

I have noticed that the install of PS has tended to fail to complete successfully, at least on a number of Debian 8 servers and instead hangs until a timeout. The systemd service also fails to start.

The issue seems to be caused by the creation of the mysql dir in datadir (https://github.com/percona/percona-server/blob/5.7/build-ps/debian/extra/mysql-systemd-start#L58).

This produces:

mysql-systemd-start[4616]: 2016-06-21T07:47:20.752695Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.

Removal of the test and creation of the directory before the initialise then works fine, although the test will produce an error (but continues) and so that needs adjusting:

mysql-systemd-start[8587]: ls: cannot access /var/lib/mysql/mysql: No such file or directory

[Tomislav Plavcic (tplavcic)]

More info from Ceri:"I have just done a PS 5.7 Deb install that seemed to go through OK - I unmounted the LV and moved the existing mount point before doing the install - perhaps the issue happens when the OS has been installed, /var/lib/mysql exists (as an LV is mounted) and possible due to ownership?"

[marki555 (marki-g)]

It happends when /var/lib/mysql is a separate filesystem and contains the lost+found directory (for example ext3/4 filesystem).
Having "ignore-db-dir=lost+found" in my.cnf works fine, but don't know if it will help also during --initialize. I had to workaround it by running the installation while having filesysten umounted and then moving the datafiles to it.

[Niranjan D Pandit (niranjan81)]

I tried ignore-db-dir=lost+found but it didn't work, in 5.7.14 path to the default datadir /var/lib/mysql is also coded in /usr/share/mysql/mysql-systemd-start as MYSQLDATA=/var/lib/mysql
changing this variable worked for me. Hope this helps.

[Fabian Niepelt (fniepelt)]

I also tried the ignore-db-dir=lost+found, however it didn't work.
The reason is that the postinst script of the package checks if the mysql datadir is empty by issuing "ls -A" on it. If it is not empty, it won't initialize the database and just skip to starting mysql.
The sanity script that is run before every mysql start will then also fail to initialize the database, because it creates the mysql directory in the datadir. But then mysqld --initialize complains that the datadir is not empty.
mysqld --initialize does honour the ignore-db-dir variable and therefor does not complain about the lost+found directory, but ignoring the mysql db dir does not seem to be a viable solution.

We're going to set the mountpoint to /var/lib instead of /var/lib/mysql as a workaround for this issue.

[Farid UYAR (fuyar)]

Hello,

I got the same issue when not using the default "/var/lib/mysql" directory as datadir.

I set the datadir in the my.cnf file to "/data/mysql" for example. Then start MySQL service and it gets stuck with the following error :

[ERROR] --initialize specified but the data directory has files in it. Aborting.

Problem is the "/usr/share/mysql/mysql-systemd-start pre" script creates a directory mysql in the datadir before launching the initialization. It should be totally empty with no files nor folders.

I have to comment out the folder creation in the pre script.

[Steve Hill (stevehill)]

I'm experiencing the same issue on a fresh Ubuntu 16.04 server.

Percona Server 5.7 starts up just fine when using the default configuration, but overriding the datadir to /mysql/mysql-data/datadir causes it to fail with the same error seen in fuyar's comment above.

Background:
1. I install Percona normally, using the percona-server-server-5.7 package.
2. I then replace the configuration file with my custom one, largely generated from the Percona Configuration Wizard.
3. I restart MySQL.
4. This then reproducibly gives the error above.

When the process fails, the datadir contains the expected ibdata* files, plus a set of *.pem files which are presumably for connecting with SSL. I can only assume this is expected!

Removing the contents of the datadir and attempting to restart MySQL has exactly the same effect.

Removing the datadir entirely causes a different error - that the datadir doesn't exist and will not be created for security reasons.

For now, I've gone back to MariaDB as this was a blocker for me using Percona.

[EvgeniyPatlan (evgeniy-patlan)]

I've just tried to reproduce the issue but without success. Could you please provide with exact version of the package which you used. Also is it enough just to set another datadir in cnf file to reproduce it?

[Steve Hill (stevehill)]

I was trying this on a Vagrant machine, using the bento/ubuntu-16.04 image.

The package version of percona-server-server-5.7 was 5.7.19-17-1.xenial.

I'm not really sure what I changed - I stripped back my install to a minimal set of packages and a default config, then added in the changes I needed a little at a time... and it now works.

It does seem that this is a known issue - I've seen blog posts commenting that a workaround is to install the percona-server-server-5.6 package first, then install the percona-server-server-5.7 package. I think it's something about the datadir paths being hardcoded in the setup scripts.

One thing that I _definitely_ changed between the initial issues I was having and getting it resolved was to set the datadir parameter with debconf before installing the package. Not sure if that will have made a difference.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3474