P5.5.5.40 Hangs after audit_log_flush set ON

[Marc]

After manual Audit Log flush initiated on Percona Server 5.5.40 hung and became unresponsive. This was a large 50+GB Audit Log file and flush was attempted using "SET GLOBAL audit_log_flush='ON':"
Nothing abnormal recorded in mysqld.log error log.
During after hang cpu, RAM, and disk utilization all normal.
PS 5.5.40 would not shutdown cleanly and required hard kill and then was stated.

After review, I believe this may be a regression related to the bugfix for bug: 1363370
https://bugs.launchpad.net/percona-server/+bug/1363370

[Marc (marcus-odell-0)]

Adding some basic information about Percona Server and Linux version:
$ mysql -V
mysql Ver 14.14 Distrib 5.5.40-36.1, for Linux (x86_64) using readline 5.1
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
uname -a
Linux [hostname removed] 2.6.18-371.el5 #1 SMP Thu Sep 5 21:21:44 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

[Sergei Glushchenko (sergei.glushchenko)]

Hi Marc,

Setting audit_log_flush to 'ON' is an instruction to close and reopen
log file (if file handler used). Fix for bug #1363370 which you
mentioned does not affect behavior of audit_log_flush. It only affects
log rotation based on size limits.

Could you please post your audit log settings here?

[Marc (marcus-odell-0)]

Hi Sergei,

Thank you. I read the patches for the bug, and it appeared to me that setting "SET GLOBAL audit_log_flush='ON';" may now result in a condition in which the log mutex is grabbed, but not released when flushing fails. This said, I my understanding of the code changes could be in error, I am not an expert C programmer.

Regarding audit log settings. All were defaults for PS 5.5.40. I simply installed/loaded the audit_log dynamically, leaving all the audit log variables to defaults. These did not persist across reboot, however I believe the variables would have been:

|audit_log_buffer_size = 1048576
|audit_log_file = audit.log
audit_log_flush = OFF
|audit_log_format = OLD
|audit_log_policy = ALL
|audit_log_rotate_on_size = 0
|audit_log_rotations = 0
|audit_log_strategy = ASYNCHRONOUS

Presently we have stopped using the audit log on our production servers until our root cause analysis is complete and we are confident the using the audit log will not cause the MySQL to hang again.

Thank you.
respectfully,
Marc ODell