RPM installer starts the service and corrupts the DB

[monty solomon]

The upstream RPM script for version 5.1 contains logic to determine if the installed server should be started.

The Percona RPM script for version 5.1 is missing that logic and always starts the server.

As a result, an already installed DB is sometimes corrupted by Percona Server and can't be recovered.

That scenario happens when the DB is present but the my.cnf is not yet configured and Percona Server is confused about where to find the InnoDB files.

[monty solomon (monty+launchpad)]

130917 21:02:51 InnoDB: Initializing buffer pool, size = 128.0M
130917 21:02:51 InnoDB: Completed initialization of buffer pool
130917 21:02:51 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 128 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
130917 21:02:52 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 128 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Progress in MB: 100
130917 21:02:53 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130917 21:02:53 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130917 21:02:53 InnoDB: Error: page 4 log sequence number 24128906350
InnoDB: is in the future! Current system log sequence number 8068106764.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: for more information.

[monty solomon (monty+launchpad)]

130917 21:02:53 Percona XtraDB (http://www.percona.com) 5.1.71-14.9 started; log sequence number 8068106764
130917 21:02:53 [Note] Event Scheduler: Loaded 0 events
130917 21:02:53 [Note] /usr/sbin/mysqld: ready for connections.

[Nilnandan Joshi (nilnandan-joshi)]

Hi,

Can you provide some more information like how exactly we can reproduce it? which exact logic is missing, where data files are located (/var/lib/mysql)? How running server was started and exact version of upstream? It would be helpful to confirm this.

Thanks.

[monty solomon (monty+launchpad)]

Download upstream MySQL-server-community-5.1.71-1.rhel5.x86_64.rpm

dump the RPM script. look for SERVER_TO_START variable and logic.

rpm -qp --scripts MySQL-server-community-5.1.71-1.rhel5.x86_64.rpm

dump the RPM script from Percona 5.1.71. look for SERVER_TO_START variable and logic. observe that it is missing.

rpm -qp --scripts Percona-Server-server-51-5.1.71-rel14.9.589.rhel6.x86_64.rpm

[monty solomon (monty+launchpad)]

# uname -a
Linux monty 2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
CentOS release 6.3 (Final)

Install Percona Server 5.1.71

# yum -y install Percona-Server-client-51 Percona-Server-devel-51 Percona-Server-server-51 Percona-Server-shared-51 Percona-Server-shared-compat-51

Notice the messages

"Starting MySQL (Percona Server)...[ OK ]
Giving mysqld 2 seconds to start

List the installed components

# rpm -qa | grep ^Percona
Percona-Server-shared-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-shared-compat-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-client-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-server-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-devel-51-5.1.71-rel14.9.589.rhel6.x86_64

Check to see that Percona is running

# service mysql status
MySQL running (6623) [ OK ]

Stop Percona

# service mysql stop
Shutting down MySQL. [ OK ]

Remove Percona

# rpm -e --nodeps `rpm -qa | grep ^Percona`

Verify removal

# rpm -qa | grep ^Percona

Download and install MySQL 5.1

# yum -y install MySQL-client-community-5.1.71-1.rhel5.x86_64.rpm MySQL-devel-community-5.1.71-1.rhel5.x86_64.rpm MySQL-server-community-5.1.71-1.rhel5.x86_64.rpm MySQL-shared-community-5.1.71-1.rhel5.x86_64.rpm MySQL-shared-compat-5.1.71-1.rhel5.x86_64.rpm

Notice that there are no messages about starting MySQL

List the installed components

# rpm -qa | grep ^MySQL
MySQL-shared-community-5.1.71-1.rhel5.x86_64
MySQL-client-community-5.1.71-1.rhel5.x86_64
MySQL-shared-compat-5.1.71-1.rhel5.x86_64
MySQL-server-community-5.1.71-1.rhel5.x86_64
MySQL-devel-community-5.1.71-1.rhel5.x86_64

Verify that MySQL is NOT running

# service mysql status
MySQL is not running [FAILED]

Remove MySQL 5.1

# rpm -e --nodeps `rpm -qa | grep ^MySQL`

Install Percona Server 5.1.71

# yum -y install Percona-Server-client-51 Percona-Server-devel-51 Percona-Server-server-51 Percona-Server-shared-51 Percona-Server-shared-compat-51

Notice the messages

"Starting MySQL (Percona Server)...[ OK ]
Giving mysqld 2 seconds to start

List the installed components

# rpm -qa | grep ^Percona
Percona-Server-shared-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-shared-compat-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-client-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-server-51-5.1.71-rel14.9.589.rhel6.x86_64
Percona-Server-devel-51-5.1.71-rel14.9.589.rhel6.x86_64

Check to see that Percona is running

# service mysql status
MySQL running (9609) [ OK ]

[monty solomon (monty+launchpad)]

Were you able to compare the RPM scripts to see the missing logic?

[Nilnandan Joshi (nilnandan-joshi)]

Yes Monty, I have checked and found below logic is missing in Percona RPM (5.1.71-rel14.9.589)

if [ -n "$SERVER_TO_START" ] ; then
                # There is only one PID file, race possibility ignored
                echo "PID file:" >> $STATUS_FILE
                ls -l $mysql_datadir/*.pid >> $STATUS_FILE
                cat $mysql_datadir/*.pid >> $STATUS_FILE
                echo >> $STATUS_FILE
                echo "Server process:" >> $STATUS_FILE
                ps -fp `cat $mysql_datadir/*.pid` >> $STATUS_FILE
                echo >> $STATUS_FILE
                echo "SERVER_TO_START=$SERVER_TO_START" >> $STATUS_FILE
        else
                # Take a note we checked it ...
                echo "PID file:" >> $STATUS_FILE
                ls -l $mysql_datadir/*.pid >> $STATUS_FILE 2>&1
        fi

if [ -f $STATUS_FILE ] ; then
        SERVER_TO_START=`grep '^SERVER_TO_START=' $STATUS_FILE | cut -c17-`
else
        SERVER_TO_START='true' # This is for 5.1 only, to not change behavior
fi

if [ "$SERVER_TO_START" = "true" ] ; then
        # Restart in the same way that mysqld will be started normally.
        if [ -x /etc/init.d/mysql ] ; then
                /etc/init.d/mysql start
                echo "Giving mysqld 2 seconds to start"
                sleep 2
        fi

        # Allow mysqld_safe to start mysqld and print a message before we exit
        sleep 2
fi

[Nilnandan Joshi (nilnandan-joshi)]

But that logic is already in 5.5.33 and 5.6.13 rpms.

[monty solomon (monty+launchpad)]

Changes to the upstream RPMs don't appear to be tracked and added to the Percona RPMs.

For example, the change was made in version 5.1.49 for MySQL bug #27072 in 2010.

How does Percona decide to incorporate upstream MySQL bug fixes?

Changes in MySQL 5.1.49 (2010-07-09)
https://dev.mysql.com/doc/relnotes/mysql/5.1/en/news-5-1-49.html

[Ignacio Nin (ignacio-nin)]

@Monty,

> How does Percona decide to incorporate upstream MySQL bug fixes?

For packaging, on a case-by-case basis. Our packaging at this point differs quite a bit from upstream's so we don't merge everything that comes in on this front.

[monty solomon (monty+launchpad)]

Did a fix make it into 5.1.72?

Thanks.

[monty solomon (monty+launchpad)]

Why was this bug changed to unassigned?

[monty solomon (monty+launchpad)]

Will this be fixed in 5.1.73?

[Valerii Kravchuk (valerii-kravchuk)]

Bug was unassigned as Ignacio does not work with us any more unfortunately. I do not know about 5.1.73, it needs assignee and fix first, before deciding on version the fix is going to.

[monty solomon (monty+launchpad)]

FYI, upstream 5.1.73 was released yesterday.

https://dev.mysql.com/doc/relnotes/mysql/5.1/en/news-5-1-73.html

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3027