Comment 9 for bug 1729536

Revision history for this message
Eric Fjøsne (efj) wrote :

@ChristianEhrhardt
Thanks for your replies and initiatives. Much appreciated.
I did indeed notice the install package we could rollback to using apt-get install mysql-server=XXX was 5.7.11, but this version is more than a year and a half old, which doesn't sound like such a good idea.

Coming back to our use case: the replication master is still running version 5.7.19 and is working flawlessly (fortunately ...), only the replication slave is running version 5.7.20. Without proof of any kind so far, but based on Olaf's message, this would suggest that regression was indeed linked to the update from 5.7.19 to 5.7.20.

As silly as it may sound, the crash is now "stabilised" on our infrastructure and handled on a daily basis using the same recovery method: fast forward the replication (basically replay the statements that weren't rolled back at the moment of crash) until there is no more replication error. I believe we will keep on doing this until a fix is published.

@Olaf:
>> What I do know though is that it has to be somehow reproducible because the timeframe within which it happens is the same every day.
> That's interesting. What makes this timeframe different from other timeframes? Are you altering tables?

It is the only time of the day (pre-production activity) when we are running DDL queries (OPTIMIZE / TRUNCATE). To my knowledge of InnoDB, OPTIMIZE is not supported as such by InnoDB, and actually performs a recreate of the whole table from scratch. TRUNCATE will also perform a full table recreate.

As for the rest of the day (actual production), we have only DML queries running on the servers, except for the upgrades we manually perform.

To move forward and get some concrete information, I will go through all of our batches and:
- disable exceptionally all the OPTIMIZE queries
- replace the TRUNCATE statements by DELETE FROM statements.

I will post some feedback here tomorrow morning.

Thanks again for your replies,

Eric