MariaDB: mysql_install_db causes server segfault

[Sergey Petrunia]

I've tried installing MariaDB from
http://master.ourdelta.org/deb/dists/jaunty/mariadb-ourdelta/ on Ubuntu 9.04 amd64.
I got HTTP 404 errors on the first attempt, but not on the second.

The package asked me to change root password several times during the
installation. In all cases, I've just hit 'enter' as an indication that I do
not want to change it.

As a result, I got this:

Setting up mariadb-server-core-5.1 (5.1.38-ourdelta81) ...
Setting up mariadb-server-5.1 (5.1.38-ourdelta81) ...
 * Stopping MariaDB database server mysqld [ OK ]
Replacement succeeded for "/usr/sbin/mysqld".
 * Starting MariaDB database server mysqld [ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
ERROR 1045 (28000): Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)

Setting up libhtml-template-perl (2.9-1) ...
Setting up mariadb-server (5.1.38-ourdelta81) ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
root@pylon64:~#
root@pylon64:~# /etc/init.d/mysql status
/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)'
 *
root@pylon64:~# /etc/init.d/mysql start
 * Starting MariaDB database server mysqld [ OK ]
root@pylon64:~#

Further experimentation showed that "/etc/init.d/mysql start" works, while
restart and stop commands do not because of permission errors.

/etc/mysql/debian.cnf contains the following:

[client]
host = localhost
user = debian-sys-maint
password = G4dQ6VLVrD81ilks
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = debian-sys-maint
password = G4dQ6VLVrD81ilks
socket = /var/run/mysqld/mysqld.sock
basedir = /usr

This user/password pair is not accepted by the server. I had to edit
/etc/my.cnf and put there 'skip-grant-tables' so that I could log in.

After logging in, I can see that mysql.user table is empty.

It looks like installer script wasn't successful in adding the
debian-stys-maint user?

[Sergey Petrunia (sergefp)]

> The package asked me to change root password several times during the
> installation. In all cases, I've just hit 'enter' as an indication that I do
> not want to change it.
And when I re-install (having removed the datadir manually) and do specify the
password when asked, I first get a dialog box saying

---
Unable to set password for the MySQL "root" user
An error occurred while setting the password for the MySQL administrative user. This may have happened because the
account already has a password, or because of a communication problem with the MySQL server.

You should check the account's password after the package installation.

Please read the /usr/share/doc/mariadb-server-5.1/README.Debian file for more information.
--

and then this output:

Setting up mariadb-server-core-5.1 (5.1.38-ourdelta81) ...
Setting up mariadb-server-5.1 (5.1.38-ourdelta81) ...
 * Stopping MariaDB database server mysqld [ OK ]
091015 2:11:06 [Note] Plugin 'FEDERATED' is disabled.
091015 2:11:06 [ERROR] mysqld: File '/var/lib/mysql/maria_log_control' not found (Errcode: 13)
091015 2:11:06 [ERROR] mysqld: Got error 'Can't open file' when trying to use maria control file '/var/lib/mysql/maria_log_control'
091015 2:11:06 [ERROR] Plugin 'MARIA' init function returned error.
091015 2:11:06 [ERROR] Plugin 'MARIA' registration as a STORAGE ENGINE failed.
091015 02:11:06 [Note] PrimeBase XT (PBXT) Engine 1.0.08d RC loaded...
091015 02:11:06 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
091015 2:11:07 InnoDB: highest supported file format is Barracuda.
091015 2:11:07 InnoDB Plugin 1.0.3-6a started; log sequence number 46409
091015 2:11:07 [ERROR] mysqld: Can't find file: './mysql/user.frm' (errno: 13)
ERROR: 1017 Can't find file: './mysql/user.frm' (errno: 13)
091015 2:11:07 [ERROR] Aborting

091015 2:11:07 InnoDB: Starting shutdown...
091015 2:11:08 InnoDB: Shutdown completed; log sequence number 46419
091015 02:11:08 [Note] PrimeBase XT Engine shutdown...
091015 2:11:08 [Note] /usr/sbin/mysqld: Shutdown complete

Replacement succeeded for "/usr/sbin/mysqld".
 * /etc/init.d/mysql: WARNING: /etc/mysql/my.cnf cannot be read. See README.Debian.gz
 * Starting MariaDB database server mysqld
                                                                                                                    [fail]
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing mariadb-server-5.1 (--configure):
 subprocess post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mariadb-server:
 mariadb-server depends on mariadb-server-5.1; however:
  Package mariadb-server-5.1 is not configured yet.
dpkg: error processing mariadb-server (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc6 ...
No ...

[Sergey Petrunia (sergefp)]

Link to inital mail-list discussion: https://lists.launchpad.net/maria-developers/msg01233.html

[Cafuego (cafuego)]

Right. The problem is that mysql_install_db never gets around to creating the initial root user; the server segfaults when it processes mysql_system_tables_data.sql.

mysql> CREATE TEMPORARY TABLE tmp_user LIKE user;
Query OK, 0 rows affected (0.00 sec)

mysql> set @current_hostname= @@hostname;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TEMPORARY TABLE tmp_db LIKE db;
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO db SELECT * FROM tmp_db WHERE @had_db_table=0;
ERROR 2013 (HY000): Lost connection to MySQL server during query

I'm building a newer package at the moment from lp:~maria-captains/maria/maria-5.1-merge revno 2788. We'll see if the problem persists.

[Sergey Petrunia (sergefp)]

Can we get any details about how the crash happens? Does MySQL print anything while crashing?

Is it possible to reproduce the crash manually? (by taking the tarball and running certain commands?) I've tried to reproduce but was not successful

[Sergey Petrunia (sergefp)]

Ok, it is possible to reproduce:
* install the packages (it doesn't matter whether the configuration fails or succeeds)
* replace /usr/sbin/mysqld with one with debug info
* comment out the "log-slow-verbosity=..." in /etc/mysql/my.cnf (No idea why, but this option causes error for our binaries. Does OurDelta apply some patches when building?)
* run mysql_install_db, or this:

echo "use test;" > /tmp/sql1
cat /usr/share/mysql/mysql_system_tables.sql /usr/share/mysql/mysql_system_tables_data.sql >> /tmp/sql1

/usr/sbin/mysqld --bootstrap --basedir=/usr --datadir=/var/lib/mysql --log-warnings=0 --loose-skip-innodb --loose-skip-ndbcluster --max_allowed_packet=8M --default-storage-engine=myisam --net_buffer_length=16K < /tmp/sql1

[Sergey Petrunia (sergefp)]

Stack trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f83fa2dd950 (LWP 1568)]
memset () at ../sysdeps/x86_64/memset.S:631
(gdb) wher
#0 memset () at ../sysdeps/x86_64/memset.S:631
#1 0x00000000008bfcd2 in myisamchk_init (param=0x7f83fa2ba990) at mi_check.c:91
#2 0x00000000008912fc in ha_myisam::enable_indexes (this=0x3682500, mode=2) at ha_myisam.cc:1469
#3 0x000000000088bff8 in ha_myisam::end_bulk_insert (this=0x3682500, abort=false) at ha_myisam.cc:1607
#4 0x000000000071e108 in handler::ha_end_bulk_insert (this=0x3682500, abort=false) at handler.h:1242
#5 0x000000000071624d in select_insert::send_eof (this=0x3b65e48) at sql_insert.cc:3241
#6 0x000000000070f87a in return_zero_rows (join=0x3bb9280, result=0x3b65e48, tables=0x3b65818, fields=@0x3aec548, send_row=false,
    select_options=3489942016, info=0xc94da8 "Impossible WHERE", having=0x0) at sql_select.cc:7225
#7 0x000000000070ff4d in JOIN::exec (this=0x3bb9280) at sql_select.cc:1735
#8 0x000000000070c9ef in mysql_select (thd=0x3aea3a8, rref_pointer_array=0x3aec610, tables=0x3b65818, wild_num=1,
    fields=@0x3aec548, conds=0x3b65cc8, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=3489942016,
    result=0x3b65e48, unit=0x3aec018, select_lex=0x3aec440) at sql_select.cc:2425
#9 0x0000000000711fdb in handle_select (thd=0x3aea3a8, lex=0x3aebf78, result=0x3b65e48, setup_tables_done_option=1073741824)
    at sql_select.cc:278
#10 0x0000000000681d79 in mysql_execute_command (thd=0x3aea3a8) at sql_parse.cc:3239
#11 0x0000000000687cdc in mysql_parse (thd=0x3aea3a8, inBuf=0x3b65290 "INSERT INTO db SELECT * FROM tmp_db WHERE @had_db_table=0",
    length=57, found_semicolon=0x7f83fa2dd0e0) at sql_parse.cc:5979
#12 0x000000000068a458 in handle_bootstrap (arg=0x3aea3a8) at sql_parse.cc:516
#13 0x00007f84118463ba in start_thread (arg=<value optimized out>) at pthread_create.c:297
#14 0x00007f84108b9fcd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()
...
(gdb) p join->thd->query
$11 = 0x3b65290 "INSERT INTO db SELECT * FROM tmp_db WHERE @had_db_table=0"

[Sergey Petrunia (sergefp)]

The fix has two parts:

https://lists.launchpad.net/maria-developers/msg01281.html (this has been pushed into lp:~maria-captains/maria/maria-5.1-merge)

https://lists.launchpad.net/maria-developers/msg01282.html (this will need to be pushed on ourdelta side)