Fake fatal error 1236 from master brakes replication on slave

[Bogdan]

Hello.

I've got following error twice on my ~17 slaves host at same time.

I'm sure that I had at least 300Gb free space for mysql logs on master and at least 100Gb space for relay logs on slaves. File system - ext4.

CentOS 6.6 amd64 and Percona server 5.6.23-rel72.1 on both master and slave. After first failure I set "sync_binlog = 1" on master, but this did not helps, second time it fails this friday. Zabbix check free space on master every 300 seconds, I don't think that hundreds of gigabytes could be really exhausted and freed between two check (there are RAID10 with BBU of four disc for logs). I don't have inode monitoring for this filesystem.

The only probably reason for 'out of space' is tmpfs of 2Gb, which is used as tmpdir. Does Percona use tmpfs for replication somehow?

Here if similar complain from MariaDB: https://lists.launchpad.net/maria-discuss/msg01780.html

Here is exact error messages:

2015-03-13 17:54:32 14091 [ERROR] Error reading packet from server: binlog truncated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.214657' at 93047076, the last event read from '/var/log/mysql/mysql-bin.247303' at 1482130547, the last byte read from '/var/log/mysql/mysql-bin.247303' at 55970472. ( server_errno=1236)
2015-03-13 17:54:32 14091 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'binlog truncated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.214657' at 93047076, the last event read from '/var/log/mysql/mysql-bin.247303' at 1482130547, the last byte read from '/var/log/mysql/mysql-bin.247303' at 55970472.', Error_code: 1236

2015-03-27 21:08:57 3014 [ERROR] Error reading packet from server: binlog truncated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.001071' at 81546762, the last event read from '/var/log/mysql/mysql-bin.023043' at 8660904, the last byte read from '/var/log/mysql/mysql-bin.023043' at 8660923. ( server_errno=1236)
2015-03-27 21:08:57 3014 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'binlog truncated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.001071' at 81546762, the last event read from '/var/log/mysql/mysql-bin.023043' at 8660904, the last byte read from '/var/log/mysql/mysql-bin.023043' at 8660923.', Error_code: 1236

Here is part of master config:

server-id = 123
replicate-do-db = advdb
log_bin = /var/log/mysql/mysql-bin.log
max_binlog_size = 100M
expire_logs_days = 1
binlog_do_db = advdb
binlog_format = MIXED
log_slave_updates = On
sync_binlog = 1

[Bogdan (bogdar)]

Config from master server

[Sveta Smirnova (svetasmirnova)]

Thank you for the report.

There are also two Oracle MySQL bug reports with same symptoms:

http://bugs.mysql.com/bug.php?id=75746 and http://bugs.mysql.com/bug.php?id=72457

This bug reports cause binlog corruption also. Please confirm or reject if this is not so in your case. Please also send us full error log file.

[Bogdan (bogdar)]

Hello.

http://bugs.mysql.com/bug.php?id=75746 looks unrelated
http://bugs.mysql.com/bug.php?id=72457 - I think is my case ;(

I've just re-checked log and found:

On master:

2015-03-27 21:08:56 8171 [ERROR] /usr/sbin/mysqld: Sort aborted: Error writing file '/var/lib/tmpfs/MYQNMhTi' (Errcode: 28 -
 No space left on device)
2015-03-27 21:08:57 8171 [ERROR] /usr/sbin/mysqld: Sort aborted: Error writing file '/var/lib/tmpfs/MYfDFB06' (Errcode: 28 -
 No space left on device)

On slave:

2015-03-27 21:08:57 3053 [ERROR] Error reading packet from server: binlog truncated in the middle of event; consider out of
disk space on master; the first event 'mysql-bin.000900' at 40694620, the last event read from '/var/log/mysql/mysql-bin.023
043' at 8660904, the last byte read from '/var/log/mysql/mysql-bin.023043' at 8660923. ( server_errno=1236)
2015-03-27 21:08:57 3053 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'binlog trun
cated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.000900' at 40694620, the last
 event read from '/var/log/mysql/mysql-bin.023043' at 8660904, the last byte read from '/var/log/mysql/mysql-bin.023043' at
8660923.', Error_code: 1236
2015-03-27 21:08:57 3053 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.023043', position 8660904

I can give you full log file, but I would not like to publish them. If there are any private way for this - I'm ready.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3277