We recently ran into an error while upgrading from Percona Server 5.5.24 to 5.5.41 on Ubuntu Precise. When we upgraded the packages it ran the mysql_install_db script as part of the postinst configure. The mysql_install_db script writes system table ddl to the binary logs which then gets replicated to the slaves. I filed a bug with Oracle because mysql_install_db should not be writing to the binary log (https://bugs.mysql.com/bug.php?id=77122) but that is only part of the problem. The postinst should not run mysql_install_db when there is a working MySQL installation. If anything, it should run mysql_upgrade --skip-binlog-write in that case.
The following code in postinst causes the the issue:
# initiate databases. Output is not allowed by debconf :-(
# Debian: beware of the bashisms...
# Debian: can safely run on upgrades with existing databases
set +e
bash /usr/bin/mysql_install_db --user=mysql --rpm 2>&1 |
$ERR_LOGGER
if [ "$?" != "0" ]; then
echo "ATTENTION: An error has occured. More info is in the syslog!"
fi
set -e
The direct error we encountered is that there was a replication conflict as the slave got upgraded after the master and mysql_upgrade ran simultaneously with the replicated statements. The bigger problem is that mysql_install_db drops and creates the performance_schema database which then gets replicated to all slaves.
Yeah, and in postinst, the 'configure' part is getting executed during a package upgrade as well.
And it mentions in the comments
"# Debian: can safely run on upgrades with existing databases"
I don't think it's safe to run mysql_install_db on upgrades. This should be changed from the postinst and only be executed during a regular install.