a long semaphore wait

[JonathanLevin]

My Percona server crashed for the second time in two weeks.
I am not sure what the issue is, but the server becomes unresponsive and I cannot connect to it.
Here is the error log out put:

-lock on RW-latch at 0x2977a48 '&buf_pool->page_hash_latch'
a writer (thread id 140389372684032) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file buf0buf.c line 2529
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.25a-rel27.1/Percona-Server-5.5.25a-rel27.1/storage/innobase/buf/buf0buf.c line 3716
InnoDB: Warning: a long semaphore wait:
--Thread 140389378275072 has waited at dict0dict.c line 744 for 252.00 seconds the semaphore:
Mutex at 0x3da81288 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 140389352183552 has waited at buf0buf.c line 2529 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x2977a48 '&buf_pool->page_hash_latch'
a writer (thread id 140389372684032) has reserved it in mode wait exclusive
number of readers 1, waiters flag 1, lock_word: ffffffffffffffff
Last time read locked in file buf0buf.c line 2529
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.25a-rel27.1/Percona-Server-5.5.25a-rel27.1/storage/innobase/buf/buf0buf.c line 3716
InnoDB: Warning: a long semaphore wait:
--Thread 140389349787392 has waited at dict0dict.c line 744 for 241.00 seconds the semaphore:
Mutex at 0x3da81288 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream

[Raghavendra D Prabhu (raghavendra-prabhu)]

Looks like a duplicate of https://bugs.launchpad.net/percona-server/+bug/1007268 which is a bug on page_hash_latch ordering.

The workaround is to enable innodb_lazy_drop_table to 1.

[JonathanLevin (boogybo)]

I will certainly give it a try.

[Stewart Smith (stewart)]

(setting to incomplete as is probable duplicate)

[Laurynas Biveinis (laurynas-biveinis)]

We have fixed a page_hash_latch bug that results in the symptoms: bug 1026926. Its fix is scheduled for 5.5.27-28.1 release. Please let us know if you still experience any issues after upgrading. Thanks!

[Noel (noelc)]

I faced this issue in Percona 5.5.32

InnoDB: Warning: a long semaphore wait:
--Thread 1383676224 has waited at buf0buf.c line 2513 for 911.00 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c lin
e 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1401514304 has waited at buf0buf.c line 2513 for 953.00 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c lin
e 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1401248064 has waited at buf0buf.c line 2513 for 1033.0 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c line 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1382877504 has waited at buf0buf.c line 2513 for 1033.0 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c line 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1416689984 has waited at buf0buf.c line 2513 for 1026.0 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c line 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1390598464 has waited at buf0buf.c line 2513 for 961.00 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspace/percona-server-5.5-rpms/label_exp/centos5-64/target/BUILD/Percona-Server-5.5.32-rel31.0/storage/innobase/buf/buf0lru.c line 1902
InnoDB: Warning: a long semaphore wait:
--Thread 1393260864 has waited at buf0buf.c line 2513 for 1013.0 seconds the semaphore:
S-lock on RW-latch at 0x1c7781c8 '&buf_pool->page_hash_latch'
number of readers 0, waiters flag 0, lock_word: 100000
Last time read locked in file buf0buf.c line 2513
Last time write locked in file /mnt/workspac...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-346