apt-upgrading percona-server fails in post-installation

[Mrten]

I've seen this in a few releases now; the starting of the server in the final moments of the upgrade process fails:

[...]
Setting up percona-server-common (5.1.62-rel13.3-435.lucid) ...
(Reading database ... 110969 files and directories currently installed.)
Preparing to replace percona-server-server-5.1 5.1.61-rel13.2-430.lucid (using .../percona-server-server-5.1_5.1.62-rel13.3-435.lucid_amd64.deb) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Unpacking replacement percona-server-server-5.1 ...
Preparing to replace percona-server-client-5.1 5.1.61-rel13.2-430.lucid (using .../percona-server-client-5.1_5.1.62-rel13.3-435.lucid_amd64.deb) ...
Unpacking replacement percona-server-client-5.1 ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up percona-server-client-5.1 (5.1.62-rel13.3-435.lucid) ...

Setting up percona-server-server-5.1 (5.1.62-rel13.3-435.lucid) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Percona Server is distributed with several useful UDF (User Defined Function) from Maatkit.
Run the following commands to create these functions:
mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
See http://code.google.com/p/maatkit/source/browse/trunk/udf for more details
 * Starting MySQL (Percona Server) mysqld [ OK ]
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing percona-server-server-5.1 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 percona-server-server-5.1
E: Sub-process /usr/bin/dpkg returned an error code (1)

The only way around this is to edit /var/lib/dpkg/info/percona-server-server-5.1.postinst and comment out the restarting of the server:

# Automatically added by dh_installinit
if [ -x "/etc/init.d/mysql" ]; then
        update-rc.d mysql defaults 19 21 >/dev/null
# if [ -x "`which invoke-rc.d 2>/dev/null`" ]; then
# invoke-rc.d mysql start || exit $?
# else
# /etc/init.d/mysql start || exit $?
# fi
fi

Then, apt-get upgrade again, which will succeed but fail to start the server (as expected):

root@piro:/home/mrten# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]?
Setting up percona-server-server-5.1 (5.1.62-rel13.3-435.lucid) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Percona Server is distributed with several useful UDF (User Defined Function) from Maatkit.
Run the following commands to create these functions:
mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
See http://code.google.com/p/maatkit/source/browse/trunk/udf for more details

Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
root@piro:/home/mrten# /etc/init.d/mysql start

[Mrten (bugzilla-ii)]

Same thing happens if I upgrade to 5.5:

root@miho:/home/mrten# apt-get install percona-server-client- percona-server-server- percona-server-client-5.5 percona-server-server-5.5
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libhtml-template-perl
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  libaio1 libmysqlclient18 percona-server-common-5.5
Suggested packages:
  tinyca
The following packages will be REMOVED:
  percona-server-client percona-server-client-5.1 percona-server-common percona-server-server percona-server-server-5.1
The following NEW packages will be installed:
  libaio1 libmysqlclient18 percona-server-client-5.5 percona-server-common-5.5 percona-server-server-5.5
0 upgraded, 5 newly installed, 5 to remove and 0 not upgraded.
Need to get 55.8MB/57.8MB of archives.
After this operation, 24.0MB disk space will be freed.
Do you want to continue [Y/n]?
Get:1 http://repo.percona.com/apt/ lucid/main percona-server-common-5.5 5.5.23-rel25.3-240.lucid [2,740B]
Get:2 http://repo.percona.com/apt/ lucid/main percona-server-client-5.5 5.5.23-rel25.3-240.lucid [19.9MB]
Get:3 http://repo.percona.com/apt/ lucid/main percona-server-server-5.5 5.5.23-rel25.3-240.lucid [35.9MB]
Fetched 55.8MB in 4min 8s (225kB/s)
Preconfiguring packages ...
(Reading database ... 43013 files and directories currently installed.)
Removing percona-server-client ...
Removing percona-server-server ...
Removing percona-server-server-5.1 ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Removing percona-server-client-5.1 ...
Removing percona-server-common ...
Processing triggers for man-db ...
Selecting previously deselected package libaio1.
(Reading database ... 37583 files and directories currently installed.)
Unpacking libaio1 (from .../libaio1_0.3.107-3ubuntu2_amd64.deb) ...
Selecting previously deselected package libmysqlclient18.
Unpacking libmysqlclient18 (from .../libmysqlclient18_5.5.23-rel25.3-240.lucid_amd64.deb) ...
Selecting previously deselected package percona-server-common-5.5.
Unpacking percona-server-common-5.5 (from .../percona-server-common-5.5_5.5.23-rel25.3-240.lucid_amd64.deb) ...
Selecting previously deselected package percona-server-client-5.5.
Unpacking percona-server-client-5.5 (from .../percona-server-client-5.5_5.5.23-rel25.3-240.lucid_amd64.deb) ...
Selecting previously deselected package percona-server-server-5.5.
Unpacking percona-server-server-5.5 (from .../percona-server-server-5.5_5.5.23-rel25.3-240.lucid_amd64.deb) ...
Processing triggers for install-info ...
install-info: warning: no info dir entry in `/usr/share/info/mysql.info.gz'
Processing triggers for man-db ...
Setting up libaio1 (0.3.107-3ubuntu2) ...

Setting up libmysqlclient18 (5.5.23-rel25.3-240.lucid) ...

Setting up percona-server-common-5.5 (5.5.23-rel25.3-240.lucid) ...

[Mrten (bugzilla-ii)]

Argh, spoke too soon; might be related to two options that I had in my.cnf that probably aren't supported in -5.5:

suppress_log_warning_1592
skip-mrg_myisam

Will report back if this bug occurs while upgrading 5.5.

[Stewart Smith (stewart)]

This is likely the cause (suppress_log_warning_159 and skip-mrg_myisam). I'll mark this bug as invalid but feel free to reopen if it turns out not to be the case.

Thanks very much for your bug report.

[Mrten (bugzilla-ii)]

I think you misread what I wrote (or I wrote not clearly enough); these two options are valid for 5.1 and it does happen when upgrading between 5.1 versions. I'll report back when it happens when I upgrade another 5.1 to 5.5.

[Stewart Smith (stewart)]

Okay, since it's between 5.1 versions I'll reopen.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Mrten,

I tested with 5.1.65 and it seems to be going well.

Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 3 reinstalled, 0 to remove and 11 not upgraded.
Need to get 0 B/49.6 MB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 61081 files and directories currently installed.)
Preparing to replace percona-server-server-5.1 5.1.65-rel14.0-475.precise (using .../percona-server-server-5.1_5.1.65-rel14.0-475.precise_amd64.deb) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Unpacking replacement percona-server-server-5.1 ...
Preparing to replace percona-server-client-5.1 5.1.65-rel14.0-475.precise (using .../percona-server-client-5.1_5.1.65-rel14.0-475.precise_amd64.deb) ...
Unpacking replacement percona-server-client-5.1 ...
Preparing to replace percona-server-common 5.1.65-rel14.0-475.precise (using .../percona-server-common_5.1.65-rel14.0-475.precise_amd64.deb) ...
Unpacking replacement percona-server-common ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up percona-server-common (5.1.65-rel14.0-475.precise) ...
Setting up percona-server-client-5.1 (5.1.65-rel14.0-475.precise) ...
Setting up percona-server-server-5.1 (5.1.65-rel14.0-475.precise) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Percona Server is distributed with several useful UDF (User Defined Function) from Maatkit.
Run the following commands to create these functions:
mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"
See http://code.google.com/p/maatkit/source/browse/trunk/udf for more details
 * Starting MySQL (Percona Server) mysqld [ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

===========

In your case it looks like the restart failed after the server started (since I see a [OK] there), so it may be something related to /etc/mysql/debian-start (which does mysql_upgrade etc.) or so.

Also, I have seen that /etc/init.d/mysql has changed a bit.

So, can you check in the syslog about probable causes of failed init? Also, let us know if it fails with latest package.

[W. Andrew Loe III (loe)]

You have probably changed your /etc/mysql/debian-start to just "exit 0", if that script has no output, the init script (which captures and pipes that output) will exit 1.

Try changing /etc/mysql/debian-start to:
echo "debian-start: not doing anything"
exit 0

It was brought up as a bug in Debian:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=663060

[Mrten (bugzilla-ii)]

I sure have, based on a lot of comments on the 'Net. Thanks for the info, I appreciate it.

Cannot test for sure if this fixes it as there's no 5.1 upgrade. Here's hoping the fix mentioned in that bug makes it into the 5.5 init.d scripts.

Can someone please check/confirm?

[Paco (francisco-trujillo-hacha)]

Hi

I confirmed that the fix mentioned in:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=663060

it is not working for both 5.1. and 5.5.

I am using ubuntu 12.10 64 bits with percona 5.5 and I still have the problem. I can resolve the problem using the solution describe in the bug description (disable the startup script in the postscript file and upgrade). When I tried to start mysql using

service mysql start

The log say that is Fail but in fact the database is running and working perfectly.

[Valerii Kravchuk (valerii-kravchuk)]

I am not able to repeat this while upgrading from latest percona-server-server-5.1 (5.1.68...) to latest percona-server-server-5.5 (5.5.31...) on 32-but Ubuntu 12.04.2.

So, all reporters, please, check if this bug is still repeatable for you.

[Mrten (bugzilla-ii)]

I just tested upgrading from 5.5.30 to 5.5.31 and it still fails to upgrade when you put an 'exit 0' directly at the start of debian-start:

Preparing to replace percona-server-server-5.5 5.5.30-rel30.1-465.precise (using .../percona-server-server-5.5_1%3a5.5.31-rel30.3-520.precise_amd64.deb) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]
Unpacking replacement percona-server-server-5.5 ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Setting up percona-server-server-5.5 (1:5.5.31-rel30.3-520.precise) ...
 * Stopping MySQL (Percona Server) mysqld [ OK ]

 * Percona Server is distributed with several useful UDF (User Defined Function) from Percona Toolkit.
 * Run the following commands to create these functions:

 mysql -e "CREATE FUNCTION fnv1a_64 RETURNS INTEGER SONAME 'libfnv1a_udf.so'"
 mysql -e "CREATE FUNCTION fnv_64 RETURNS INTEGER SONAME 'libfnv_udf.so'"
 mysql -e "CREATE FUNCTION murmur_hash RETURNS INTEGER SONAME 'libmurmur_udf.so'"

 * See http://www.percona.com/doc/percona-server/5.5/management/udf_percona_toolkit.html for more details

 * Starting MySQL (Percona Server) database server mysqld [ OK ]
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing percona-server-server-5.5 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 percona-server-server-5.5
E: Sub-process /usr/bin/dpkg returned an error code (1)

Also, if I currently start a server part of a percona-cluster with the initscript and it's going to do an SST it claims to have failed as well, so there are still things to tweak on the init-script.

[Valerii Kravchuk (valerii-kravchuk)]

OK, so this is is with "exit 0" added on top of /etc/mysql/debian-start. Got it.

Had you tried to add:

echo "debian-start: not doing anything"
exit 0

instead as suggested in one of previous comments?

As for Percona XtraDB Cluster startup and SST, I think this is a topic for a separate bug report at https://bugs.launchpad.net/percona-xtradb-cluster. Duplicate of https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1099428 maybe?

[Mrten (bugzilla-ii)]

Or course, as a workaround it's suitable enough (still works, just verified to finish the 5.5 upgrade I just did, easier than editing post-inst as I first suggested).

I can't currently test any 5.1 > 5.5 upgrades, seeing that it's all production, sorry, but if the initscript isn't patched my guess is that the problem occurs there as well (and testing 5.1 > 5.1 isn't possible as there is no 5.1 in the percona repositories anymore as far as I could see)

Thank you for the pointer to 1099428, that is indeed what I am seeing.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-559