Problems with debian upgrade

[Hrvoje Matijakovic]

Problems reported via MPB comment http://www.mysqlperformanceblog.com/2013/01/30/announcing-percona-xtradb-cluster-5-5-29-23-7-1/#comment-1242845

Errors during apt upgrade on debian :

130131 21:28:07 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.1 started; log sequence number 68570443738
ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_v$
130131 21:28:07 [ERROR] Aborting
...

130131 21:28:25 Percona XtraDB (http://www.percona.com) 1.1.8-rel29.1 started; log sequence number 68570443738
ERROR: 1050 Table 'plugin' already exists
130131 21:28:25 [ERROR] Aborting
...

It did not prevent apt to finish the package installation but the --configure phase of dpkg does not seem completed and so apt is now always complaining about this and trying to stop/start the server to complete the upgrade ... will report this on Percona groups, this is problematic but did not prevent us to restart the server

[Laurent Minost (lolomin)]

Hi,

Would like to add additional informations as the problem is double in the logs :
- Some problems occured with upgrade from 5.5.28 to 5.5.29, these are the two error messages displayed above and now we are having apt-get which always want to finish the --configure part of dpkg, this is problematic due to the fact that it stops the server to apply changes then restart it but fails to apply changes so then give up and restart the server.

- A second problem occured this morning while the server was up and running serving requests, suddenly load average increases a lot (up to 137) and the server was in consequence very slow, unable to answer queries and a bunch of log entries were displayed (last lines of mysql-error_2.zip file attached after the server start line )

I don't know if these two problems are related or not, if it is then only this bug entry will be enough otherwise, maybe another one will be needed to separate both problems ?

At disposal to give additional informations if needed.

Regards,

Laurent

[Ignacio Nin (ignacio-nin)]

Hello

Laurent, the issues could be related, but it would seem that the package update issue is being triggered by the Abort happening on update time. All these commands (ALTER TABLE...) are performed on the live server and it could be that the server is failing while executing the commands.

Unfortunately I'm being unable to reproduce it in debian squeeze, so please I'd appreciate if you could give me more information:

Is this debian squeeze? Also, could you paste a SHOW CREATE TABLE of the plugin table? This could help me identify if the commands when updating have got anything to do with the error.

This:

ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_v$

is particulary strange, this that command (ALTER TABLE user ADD column Show_view_priv... ) is in the middle of a sequence and it's not ill-written.

Another question -- Is bash installed, or is /bin/bash a symlink to another shell, i.e. dash? That could be helpful.

Thanks!

[Laurent Minost (lolomin)]

Hi Ignacio,

Thanks for your involvement and time in this problem.

To begin, here is the current status : for the moment we did not had any more errors or strange behaviour like we had a few hours after upgrading.

The system used here is a recnetly upgraded Debian Wheezy from Squeeze :
# cat /etc/debian_version
7.0

What is sure is that I tested again to upgrade a node (PXDBC 5.5.28 -> 5.5.29) on another cluster with Wheezy on it and had the same errors so this could be probably due to Debian Wheezy ? I do not have any Debian Squeeze with PXDB installed on it to test.

Here is the SHOW CREATE TABLE plugin output :

mysql> SHOW CREATE TABLE plugin;
+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plugin | CREATE TABLE `plugin` (
  `name` varchar(64) NOT NULL DEFAULT '',
  `dl` varchar(128) NOT NULL DEFAULT '',
  PRIMARY KEY (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='MySQL plugins' |
+--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

For bash :

# which bash
/bin/bash

# ls -al /bin/bash
-rwxr-xr-x 1 root root 975488 Dec 30 02:39 /bin/bash

# dpkg -l bash
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================================-========================-========================-====================================================================================
ii bash 4.2+dfsg-0.1 amd64 GNU Bourne Again SHell

For info, I have 2 nodes which are still using PXDBC 5.5.28 so I did not upgraded yet to 5.5.29 as this is a test cluster for the moment, so if you need me to do some tests that can be useful to you, don't hesitate, I will try to help.

Thanks.

Regards,

Laurent

[Laurent Minost (lolomin)]

Hi,

Some additional notes on the things that are staying wrong after upgrade, my aptitude is trying to finish the job for the percona-xtradb-cluster-server-5.5 package everytime I ask it to upgrade/install a new package which is causing unnecessary stop/start of MySQL server :

root@galera1:~# aptitude upgrade
The following partially installed packages will be configured:
  percona-xtradb-cluster-server-5.5
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Setting up percona-xtradb-cluster-server-5.5 (5.5.29-23.7.1-387.squeeze) ...
[ ok ] Stopping MySQL (Percona XtraDB Cluster): mysqld.
[ ok ] Starting MySQL (Percona XtraDB Cluster) database server: mysqld . . . . . . ..
[info] Checking for corrupt, not cleanly closed and upgrade needing tables..
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing percona-xtradb-cluster-server-5.5 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 percona-xtradb-cluster-server-5.5
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up percona-xtradb-cluster-server-5.5 (5.5.29-23.7.1-387.squeeze) ...
[ ok ] Stopping MySQL (Percona XtraDB Cluster): mysqld.
[ ok ] Starting MySQL (Percona XtraDB Cluster) database server: mysqld . . . . . . . . ..
[info] Checking for corrupt, not cleanly closed and upgrade needing tables..
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing percona-xtradb-cluster-server-5.5 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 percona-xtradb-cluster-server-5.5

Do you know if I can safely avoid it to try finishing the configuration part of the package as it will never finish in the current case due to previous errors reported ? If yes, then how ?

Regards,

Laurent

[Laurent Minost (lolomin)]

Hi Ignacio,

Do you have any informations about why the "post-install" script failed please ?
Don't hesitate if you need any more details.

Thanks. Regards,

Laurent

[Laurent Minost (lolomin)]

Hi,

FI if the problem is concerning anyone else : I've finally found a temporary workaround to avoid apt continuing trying to do the postinst/configure part of dpkg :

- Edited /var/lib/dpkg/info/percona-xtradb-cluster-server-5.5.postinst script then commented out the following two lines which were a problem in my case :

    echo "$fix_privs" | $MYSQL_BOOTSTRAP 2>&1 | $ERR_LOGGER
    echo "$install_plugins" | $MYSQL_BOOTSTRAP 2>&1 | $ERR_LOGGER

- Ran apt/aptitude again to check, it tried to do again its configure part for the percona-xtradb-cluster-server-5.5 package which this time succeeded and terminates without error, after this I recovered a functionnal apt.

- Concerning the cause of the problem, I still not have any clear idea of what is the problem.

Regards,

Laurent

[dannie (daniz)]

Seems to be similar to the bug 1049105 where I also have the problem with postinst.

I just tried your temp solution and can conclude that it did not work for me sadly. I still have to manually change wsrep_cluster_address to gcomm://
do an apt-get upgrade (success)
change wsrep_cluster_address to point to the cluster again

[Laurent Minost (lolomin)]

Hi,

Did an upgrade from PXDBC 5.5.29 to latest release 5.5.30 today, everything was fine except the same errors that were reported for this initial bug entry, here is the complete error log entries from the beginning of the update to the final restart of the node :

http://pastebin.com/N6kMjqhy

I mention that this NOT prevents us to restart the node properly but it seems the server crashed two times because of these errors and these crashed could be avoided ...

Regards,

Laurent

[Ilias Bertsimas (d-ilias)]

Hi,

I also got the issue with the ALTER in postinstall script after upgrading PXC from 5.5.28 to 5.5.30.
Also postinstall does not wait long enough for the node to perform an IST and finish joining the cluster thinking it failed leaving the package installation in a bad state.

Kind Regards,
Ilias.

[Amol (ajkedar)]

Hi,
I did get this errors while upgrading from 5.5.28 to 5.5.30 and also from 5.5.29 to 5.5.30
on ubuntu 10.04 LTS

[Alexander Blagoveschensky (alexminsk2)]

hello.

 I have the same issue for Debian GNU/Linux 6.0.

here is dpkg.log:

2013-05-26 09:14:31 startup packages configure
2013-05-26 09:14:31 configure percona-server-server-5.5 5.5.31-rel30.3-520.squeeze 5.5.31-rel30.3-520.squeeze
2013-05-26 09:14:31 status half-configured percona-server-server-5.5 5.5.31-rel30.3-520.squeeze

here is mysql-error.log:

130526 9:14:43 InnoDB: Waiting for the background threads to start
130526 9:14:44 Percona XtraDB (http://www.percona.com) 5.5.31-rel30.3 started; log sequence number 1703407689268
ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1
130526 9:14:44 [ERROR] Aborting

130526 9:14:44 InnoDB: Starting shutdown...
130526 9:14:49 InnoDB: Shutdown completed; log sequence number 1703407689268
130526 9:14:49 [Note] /usr/sbin/mysqld: Shutdown complete
q

tried to add trap "" SIGHUP into /etc/mysql/debian-start as described here https://bazaar.launchpad.net/~knielsen/ourdelta/bug_484127_484120_2/revision/68
but doesn't help.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Most of the things the postinst runs under configure are not
required during upgrade.

Things like: (one or more in the following are run)

a) Updating the privilege tables and plugins.

b) mysql_install_db

c) Recreating debian.cnf

etc.

Based on the flowchart here - http://wiki.debian.org/MaintainerScripts?action=AttachFile&do=get&target=upgrade.png -- linked from http://wiki.debian.org/MaintainerScripts

We need to distinguish between upgrade and fresh installation by
looking at the second parameter which is the version.

Also, PXC has imported this script directly from PS. So this is a
PS bug as well.

[Raghavendra D Prabhu (raghavendra-prabhu)]

a )Regarding the SIGHUP issue I have fixed it here https://bazaar.launchpad.net/~raghavendra-prabhu/percona-xtradb-cluster/scripting-fixes/revision/392 (for next release)

b) Regarding the issue with upgrade, manually adding a check for
upgrade in configure section of the script should fix it.

c) Regarding debian-start, we have found that it is more
unhelpful than helpful. Hence I have added like

            # Now start mysqlcheck or whatever the admin wants.
             # only if the file /etc/mysql/NO-DEBIAN-START is absent.
             if ! test -e /etc/mysql/NO-DEBIAN-START;then
                 output=$(/etc/mysql/debian-start)
                 [ -n "$output" ] && log_action_msg "$output"
                fi

    in the the init code for next PXC release.

[Valerii Kravchuk (valerii-kravchuk)]

I think we have other bug report(s) for post-install script problems of PS on Debian and Ubuntu, see https://bugs.launchpad.net/percona-server/+bug/1002500

[Raghavendra D Prabhu (raghavendra-prabhu)]

Related bug - https://bugs.launchpad.net/percona-server/+bug/1002500 but it is not dealing this part of the debian init script.

[Ignacio Nin (ignacio-nin)]

It would seem that the problem causing this bug was fixed together with the foreign key issue, as detailed in comments in http://www.mysqlperformanceblog.com/2013/01/30/announcing-percona-xtradb-cluster-5-5-29-23-7-1/#comment-1242845.

Also, as of the upcoming Percona XtraDB Cluster 5.5.33, the bug seems no longer reproducible un updates to 5.5.33.

Marking as Fix Comitted for now, if you find any issues or you are able to reproduce it under some circumstances please let us know.

[Raghavendra D Prabhu (raghavendra-prabhu)]

If you see like

==============
130916 0:15:45 [ERROR] Slave SQL: Error 'Duplicate column name 'File_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user add File_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060
130916 0:15:45 [Warning] WSREP: RBR event 1 Query apply warning: 1, 532
130916 0:15:45 [Warning] WSREP: Ignoring error for TO isolated action: source: ecbbe2eb-1e36-11e3-a81b-a63f18512e2e version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 55, g: 532, s: 531, d: 531, ts: 1379270745278593506)
130916 0:15:45 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user add Grant_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060
130916 0:15:45 [Warning] WSREP: RBR event 1 Query apply warning: 1, 533
130916 0:15:45 [Warning] WSREP: Ignoring error for TO isolated action: source: ecbbe2eb-1e36-11e3-a81b-a63f18512e2e version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 56, g: 533, s: 532, d: 532, ts: 1379270745289432756)
130916 0:15:45 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE host add Grant_priv enum('N','Y') NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060
130916 0:15:45 [Warning] WSREP: RBR event 1 Query apply warning: 1, 534
130916 0:15:45 [Warning] WSREP: Ignoring error for TO isolated action: source: ecbbe2eb-1e36-11e3-a81b-a63f18512e2e version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 57, g: 534, s: 533, d: 533, ts: 1379270745291123713)
130916 0:15:45 [ERROR] Slave SQL: Error 'Duplicate column name 'Grant_priv'' on query. Default database: 'mysql'. Query: 'ALTER TABLE db add Grant_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add References_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Index_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL,add Alter_priv enum('N','Y') COLLATE utf8_general_ci NOT NULL', Error_code: 1060
130916 0:15:45 [Warning] WSREP: RBR event 1 Query apply warning: 1, 535
130916 0:15:45 [Warning] WSREP: Ignoring error for TO isolated action: source: ecbbe2eb-1e36-11e3-a81b-a63f18512e2e version: 2 local: 0 state: APPLYING flags: 65 conn_id: 9 trx_id: -1 seqnos (l: 58, g: 535, s: 534, d: 534, ts: 1379270745292659319)
130916 0:15:45 [ERROR] Slave SQL: Error 'Duplicate column name 'ssl_type'' on query. Default database: 'mysql'. Query: 'ALTER TABLE user
 ADD ssl_type enum('','ANY','X509', 'SPECIFIED') COLLATE utf8_general_ci NOT NULL,
 ADD ssl_cipher BLOB NOT NULL,
 ADD x509_issuer BLOB NOT NULL,
 ADD x509_subject BLOB NOT NULL', Error_code: 1060
130916 0:15:45 [Warning] WSREP: RBR event 1 Query apply warning: 1, 536
130916 0:15:45 [Warning] WSREP: Ignoring ...

[Valerii Kravchuk (valerii-kravchuk)]

Is anybody able to reproduce this with PS 5.5.34+ or 5.6.14+?

[Mrten (bugzilla-ii)]

I see this going from 5.5.37-25.10-756.precise to 5.5.39-25.11-816.precise.

mysqld_safe[12663]: 141005 9:54:40 Percona XtraDB (http://www.percona.com) 5.5.39-36.0 started; log sequence number 2741896304074
mysqld_safe[12663]: ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1
mysqld_safe[12663]: 141005 9:54:40 [ERROR] Aborting

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1050

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2886