Debian package ignores GTID options when restarts server

[Sveta Smirnova]

When one upgrades Percona Server using Debian package, postinstall script restarts server few times. It can lead to situation when GTIDs sequence reset.

How to repeat.

1. Install Percona Server.
2. Start it with options:

binlog_gtid_simple_recovery = 1
enforce-gtid-consistency = 1
gtid_mode = ON

3. Insert some data, got GTIDs 1-N where N is any number, greater than 1.

4. Upgrade server

5. Insert some data, notice what GTIDs were reset to 1

[Sveta Smirnova (svetasmirnova)]

See also file #5 in the associated issue for detailed steps to reproduce.

Workaround: binlog_gtid_simple_recovery = 0

[Sveta Smirnova (svetasmirnova)]

Actually option binlog_gtid_simple_recovery = 1 strictly says what it can cause GTID resets if few binary logs without GTIDs were created after last log with GTIDs was closed, but this should not happen during upgrade.

Suggested fix: do not ignore GTID-related options, set in the configuration file.

[Vlad Lesin (vlad-lesin)]

There can be several variants of fix:

1) Do not ignore GTID related options on executing upgrade scripts. But all upgrade sql statements will be executed on slaves too, and this can be not desirable.

2) Force binlog_gtid_simple_recovery = 0 on first start after upgrade. This will work but can lead to lag during first start if there are big amount of binary log records.

3) Tell somehow to the server what exactly binary log file was the last just before the upgrade. In this case we could start usual procedure of searching gtid parameters with binlog_gtid_simple_recovery = 1 but use the passed binlog file as the last binary log for searching process.

[Laurynas Biveinis (laurynas-biveinis)]

Vlad, can you point me to the source code of script of Debian upgrade?

2) can be implemented by adding such start (and shutdown) to upgrade script itself. But it may take a long time to complete as you say.

[Vlad Lesin (vlad-lesin)]

The issue is upgrade script uses server's bootstap mode to execute some queries, and the server does not write GTID's in this mode. The simplified way to repeat:

1) Start it with options:

binlog_gtid_simple_recovery = 1
enforce-gtid-consistency = 1
gtid_mode = ON

2) Insert some data, got GTIDs 1-N where N is any number, greater than 1.

3) Stop the server and then execute any command in bootstrap mode. Example:

echo "insert into test.t1 (b) select b from test.t1;" | mysqld --bootstrap --user=mysql --skip-grant-tables

4) Start the server, GTIDs will begin with 1.

[Vlad Lesin (vlad-lesin)]

static int init_server_components()
{
...
  if (gtid_mode >= 1 && opt_bootstrap)
  {
    sql_print_warning("Bootstrap mode disables GTIDs. Bootstrap mode "
    "should only be used by mysql_install_db which initializes the MySQL "
    "data directory and creates system tables.");
    gtid_mode= 0;
  }
...
}

[Laurynas Biveinis (laurynas-biveinis)]

Vlad, check 5.7:

- The 5.6 change is cfc41cd167bc36c9bca390ab5084ddf6c11e93b8, "BUG#13992602: BINLOGGING STATEMENTS WITH GTID CRASHES IN BOOTSTRAP MODE"

- This was changed in 5.7: f2300ce39b97c66dc022a4396e00b9b27ebce878, "BUG#20980271: FIX GTID MODE TO WORK CORRECTLY IN --BOOTSTRAP/--INITIALIZE MODE"

The 5.7 patch is not merely removing the startup flag change, but more involved. At least 5.7 can be checked from user point of view whether upgrade in bootstrap mode is binlogged and not replicated OK, then a backport feasibility can be evaluated.

[Vlad Lesin (vlad-lesin)]

> At least 5.7 can be checked from user point of view whether upgrade in bootstrap mode is binlogged and not replicated OK, then a backport feasibility can be evaluated.

5.7 does not use bootstrap for upgrade, it uses --initialize-insecure mode instead.

[Laurynas Biveinis (laurynas-biveinis)]

20980271 suggests that GTIDs in 5.7 work in initialize mode the same as in bootstrap mode.

[Tomislav Plavcic (tplavcic)]

PR 5.6: https://github.com/percona/percona-server/pull/366

[Laurynas Biveinis (laurynas-biveinis)]

5.7?

[Tomislav Plavcic (tplavcic)]

5.7 shouldn't be affected as it doesn't run those scripts on upgrade, only on first install

[Laurynas Biveinis (laurynas-biveinis)]

OK, please update the bug status accordingly. Does it need a null merge to 5.7?

[Tomislav Plavcic (tplavcic)]

sorry updated the bug status, null merge was done before

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3317