Percona Server 5.7.10-3 got signal 11 with pfs_end_temp_file_open_wait_and_bind_to_descriptor_v1 on top of stack

[Baffaleuf]

+--------------+
| version() |
+--------------+
| 5.7.10-3-log |
+--------------+

mysqld partially crashed (process still active and stopped manually by signal 15)

error output :

05:11:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=131072
max_used_connections=26
max_threads=201
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1128190 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x658c543b5660
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 658d8ee57e70 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2e)[0xfe954e]
mysqld(handle_fatal_signal+0x4a1)[0x93ecb1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x65932f6678d0]
mysqld(pfs_end_temp_file_open_wait_and_bind_to_descriptor_v1+0x19)[0x12e2e89]
mysqld(real_open_cached_file+0xbc)[0xfd25ac]
mysqld(my_b_flush_io_cache+0x300)[0xfd4da0]
mysqld(_my_b_write+0x60)[0xfd5250]
mysqld(_Z12mysql_updateP3THDR4ListI4ItemES4_y15enum_duplicatesPyS6_+0x195a)[0xe8086a]
mysqld(_ZN14Sql_cmd_update23try_single_table_updateEP3THDPb+0x19c)[0xe8197c]
mysqld(_ZN14Sql_cmd_update7executeEP3THD+0x2c)[0xe8274c]
mysqld(_Z21mysql_execute_commandP3THDb+0x1c68)[0xe06d38]
mysqld(_Z11mysql_parseP3THDP12Parser_state+0x50d)[0xe0a7ed]
mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1255)[0xe0bb65]
mysqld(_Z10do_commandP3THD+0x1cd)[0xe0c33d]
mysqld(handle_connection+0x27c)[0xebf45c]
mysqld(pfs_spawn_thread+0x1b4)[0x12ded14]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x65932f6600a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x65932d8ab62d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (658c547677b0): update db_temp.work_ORA_SONYMUSIC_releve_31895 set tablespace_name = ltrim(rtrim(tablespace_name))
Connection ID (thread ID): 43371198
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[Sveta Smirnova (svetasmirnova)]

Thank you for the report.

Please provide output of SHOW CREATE TABLE db_temp.work_ORA_SONYMUSIC_releve_31895; and SHOW TABLE STATUS LIKE 'work_ORA_SONYMUSIC_releve_31895';

[Baffaleuf (dbaffaleuf)]

Dear Sveta,

Thanks for your answer,

mysql> show table status like 'work_ORA_SONYMUSIC_releve_31895' \G
*************************** 1. row ***************************
           Name: work_ORA_SONYMUSIC_releve_31895
         Engine: InnoDB
        Version: 10
     Row_format: Dynamic
           Rows: 406
 Avg_row_length: 121
    Data_length: 49152
Max_data_length: 0
   Index_length: 0
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2017-02-26 06:11:02
    Update_time: 2017-02-27 09:19:37
     Check_time: NULL
      Collation: latin1_general_cs
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.00 sec)

mysql> show create table work_ORA_SONYMUSIC_releve_31895 \G
*************************** 1. row ***************************
       Table: work_ORA_SONYMUSIC_releve_31895
Create Table: CREATE TABLE `work_ORA_SONYMUSIC_releve_31895` (
  `date_sample` datetime NOT NULL,
  `tablespace_name` varchar(255) COLLATE latin1_general_cs NOT NULL,
  `total_data` mediumint(8) unsigned NOT NULL,
  `free_data` mediumint(8) unsigned NOT NULL,
  `nb_datafiles` mediumint(8) unsigned NOT NULL,
  `max_size` mediumint(8) unsigned NOT NULL,
  PRIMARY KEY (`date_sample`,`tablespace_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs
1 row in set (0.00 sec)

Thanks,

[Baffaleuf (dbaffaleuf)]

Dear support

Same occurence during the weekend, this time the entire mysqld address space crash with signal 11:

Stack:

08:11:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=131072
max_used_connections=27
max_threads=201
thread_count=7
connection_count=7
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1128190 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x6c8d6c3a82e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 6c8e1b6c3e70 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2e)[0xfe954e]
mysqld(handle_fatal_signal+0x4a1)[0x93ecb1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x6c93c67b18d0]
mysqld(pfs_end_temp_file_open_wait_and_bind_to_descriptor_v1+0x19)[0x12e2e89]
mysqld(real_open_cached_file+0xbc)[0xfd25ac]
mysqld(my_b_flush_io_cache+0x300)[0xfd4da0]
mysqld(_my_b_write+0x60)[0xfd5250]
mysqld(_Z12mysql_updateP3THDR4ListI4ItemES4_y15enum_duplicatesPyS6_+0x195a)[0xe8086a]
mysqld(_ZN14Sql_cmd_update23try_single_table_updateEP3THDPb+0x19c)[0xe8197c]
mysqld(_ZN14Sql_cmd_update7executeEP3THD+0x2c)[0xe8274c]
mysqld(_Z21mysql_execute_commandP3THDb+0x1c68)[0xe06d38]
mysqld(_Z11mysql_parseP3THDP12Parser_state+0x50d)[0xe0a7ed]
mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1255)[0xe0bb65]
mysqld(_Z10do_commandP3THD+0x1cd)[0xe0c33d]
mysqld(handle_connection+0x27c)[0xebf45c]
mysqld(pfs_spawn_thread+0x1b4)[0x12ded14]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x6c93c67aa0a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x6c93c49f562d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (6c8d6d453d60): update db_temp.work_ORA_SONYMUSIC_releve_14411 set tablespace_name = ltrim(rtrim(tablespace_name))
Connection ID (thread ID): 48055155
Status: NOT_KILLED

Any status on your side ?

Thanks,

[Baffaleuf (dbaffaleuf)]

Hi,

Third occurence last saturday, this time on a LOAD DATA INFILE :

20:22:05 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=131072
max_used_connections=19
max_threads=201
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1128190 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7686e5fc69e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7687570efe70 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2e)[0xfe954e]
mysqld(handle_fatal_signal+0x4a1)[0x93ecb1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x768cee9e38d0]
mysqld(pfs_end_temp_file_open_wait_and_bind_to_descriptor_v1+0x19)[0x12e2e89]
mysqld(real_open_cached_file+0xbc)[0xfd25ac]
mysqld(my_b_flush_io_cache+0x300)[0xfd4da0]
mysqld(_my_b_write+0x60)[0xfd5250]
mysqld(_ZN9Log_event23wrapper_my_b_safe_writeEP11st_io_cachePKhm+0x54)[0xf6e344]
mysqld(_ZN14Rows_log_event15write_data_bodyEP11st_io_cache+0xd8)[0xf6e6a8]
mysqld(_ZN9Log_event5writeEP11st_io_cache+0x5c)[0xf82ffc]
mysqld(_ZN17binlog_cache_data11write_eventEP3THDP9Log_event+0x28)[0xf888b8]
mysqld(_ZN13MYSQL_BIN_LOG32flush_and_set_pending_rows_eventEP3THDP14Rows_log_eventb+0x60)[0xf8bd60]
mysqld(_ZN3THD33binlog_prepare_pending_rows_eventI20Write_rows_log_eventEEP14Rows_log_eventP5TABLEjmbPT_PKh+0x93)[0xf97d53]
mysqld(_ZN3THD16binlog_write_rowEP5TABLEbPKhS3_+0xc0)[0xf90740]
mysqld(_ZN20Write_rows_log_event27binlog_row_logging_functionEP3THDP5TABLEbPKhS5_+0x12)[0x99ec72]
mysqld(_Z14binlog_log_rowP5TABLEPKhS2_PFbP3THDS0_bS2_S2_E+0x27c)[0x99be7c]
mysqld(_Z12write_recordP3THDP5TABLEP9COPY_INFOS4_+0x181)[0xf484c1]
mysqld(_Z10mysql_loadP3THDP12sql_exchangeP10TABLE_LISTR4ListI4ItemES8_S8_15enum_duplicatesb+0x1d04)[0xf53ba4]
mysqld(_Z21mysql_execute_commandP3THDb+0xdc7)[0xe05e97]
mysqld(_Z11mysql_parseP3THDP12Parser_state+0x50d)[0xe0a7ed]
mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1255)[0xe0bb65]
mysqld(_Z10do_commandP3THD+0x1cd)[0xe0c33d]
mysqld(handle_connection+0x27c)[0xebf45c]
mysqld(pfs_spawn_thread+0x1b4)[0x12ded14]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x768cee9dc0a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x768cecc2762d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7686903a67d0): LOAD DATA INFILE '/home/infogerance/db_repo/tmp/db_temp.work_MS_BLABLABLA_releve_9435.tmp' REPLACE INTO TABLE db_temp.work_MS_BLABLABLA_releve_9435 FIELDS TERMINATED BY ','
Connection...

[Baffaleuf (dbaffaleuf)]

Dear support, any news on this support request ?

Thanks,

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3653