Performance Schema MD5sum abort in RHEL FIPS=1 mode

[Nickolay Ihalainen]

Stack trace from percona xtradb cluster 5.6.37-82.2-56-log
Thread pointer: 0x26e8a80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f23fc51ccf0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x914abb]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x67eef1]
/usr/lib64/libpthread.so.0(+0xf5e0)[0x7f242b6795e0]
/usr/lib64/libc.so.6(gsignal+0x37)[0x7f24297821f7]
/usr/lib64/libc.so.6(abort+0x148)[0x7f24297838e8]
/usr/lib64/libcrypto.so.10(+0x6d98f)[0x7f242a5a198f]
/usr/lib64/libcrypto.so.10(MD5_Init+0x49)[0x7f242a5a8209]
/usr/sbin/mysqld(compute_md5_hash+0x36)[0x923996]
/usr/sbin/mysqld(_Z21find_or_create_digestP10PFS_threadPK18sql_digest_storagePKcj+0x8f)[0x989b3f]
/usr/sbin/mysqld[0x9b43ef]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x3b7)[0x70f737]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x2d1)[0x712db1]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x182)[0x6d91e2]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6d9410]
/usr/sbin/mysqld(pfs_spawn_thread+0x146)[0x9b0be6]
/usr/lib64/libpthread.so.0(+0x7e25)[0x7f242b671e25]
/usr/lib64/libc.so.6(clone+0x6d)[0x7f242984534d]

How to reproduce:
Setup new centos 7 instance https://app.vagrantup.com/centos/boxes/7
Setup FIPS: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sect-security_guide-federal_standards_and_regulations-federal_information_processing_standard
do not forget to add ips=1 boot=/dev/sda2 to the end of GRUB_CMDLINE_LINUX line located in /etc/sysconfig/grub and generate new config with grub2-mkconfig -o /boot/grub2/grub.cfg

Execute any statement against mysql server 5.6 or 5.7 compiled against openssl with performance schema enabled.

[Laurynas Biveinis (laurynas-biveinis)]

Upstream?

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1132