| 2014-04-17 22:50:30 |
Chehai Wu |
bug |
|
|
added bug |
| 2014-04-17 23:40:43 |
Alan Stebbens |
bug |
|
|
added subscriber Alan Stebbens |
| 2014-04-18 16:51:12 |
Chehai Wu |
summary |
Table reference counter is not decreased in wsrep foreign key check |
"DROP DATABASE" may block indefinitely |
|
| 2014-04-18 16:56:26 |
Chehai Wu |
description |
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever. |
Steps to Reproduce:
1, wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster-56/LATEST/source/Percona-XtraDB-Cluster-5.6.15.tar.gz
2, tar xzvf Percona-XtraDB-Cluster-5.6.15.tar.gz
3, cd Percona-XtraDB-Cluster-5.6.15
4, cmake .
5, make
6, cd
7, Install mysqlsandbox (http://mysqlsandbox.net/index.html)
8, make_sandbox_from_source ~/Percona-XtraDB-Cluster-5.6.15/ replication
9, Go to the sandbox directory (~/sandboxes/rsandbox_mysql-5_6_15 on my machine).
10, Import the attached sql file into master (./m < dbs.sql).
11, ./stop_all
12. Add "table_definition_cache = 400" and "innodb_open_files = 400" to master and slave's my.cnf (master/my.sandbox.cnf, node1/my.sandbox.cnf, and node2/my.sandbox.cnf)
13, ./start_all
14, ./m -e "DELETE FROM db444.t; DROP DATABASE db444"
15, If the command in Step 14 blocks, the bug is reproduced on master. If the above command finishes, run "./s1 -e 'show slave status\G' and "./s2 -e 'show slave status\G'". If one slave stucks in "System lock" state, the bug is reproduced on the slave. If both slaves are OK, go back to Step 14 and try a different database (db1, db2, ... db999) until the bug is reproduced.
Possible Explanation:
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever.
Possible Fix:
I fixed my copy of PXC by applying the patch attached here. I am not very familiar with MySQL source code, so I am not sure if my fix is the right way to fix the problem or not. But, I cannot reproduce the problem any more after the fix. |
|
| 2014-04-18 16:57:05 |
Chehai Wu |
attachment added |
|
1000 Databases https://bugs.launchpad.net/codership-mysql/+bug/1309241/+attachment/4087724/+files/dbs.sql |
|
| 2014-04-18 16:58:11 |
Chehai Wu |
attachment added |
|
Possible Fix https://bugs.launchpad.net/codership-mysql/+bug/1309241/+attachment/4087725/+files/my_fix.diff |
|
| 2014-04-18 17:21:18 |
Chehai Wu |
description |
Steps to Reproduce:
1, wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster-56/LATEST/source/Percona-XtraDB-Cluster-5.6.15.tar.gz
2, tar xzvf Percona-XtraDB-Cluster-5.6.15.tar.gz
3, cd Percona-XtraDB-Cluster-5.6.15
4, cmake .
5, make
6, cd
7, Install mysqlsandbox (http://mysqlsandbox.net/index.html)
8, make_sandbox_from_source ~/Percona-XtraDB-Cluster-5.6.15/ replication
9, Go to the sandbox directory (~/sandboxes/rsandbox_mysql-5_6_15 on my machine).
10, Import the attached sql file into master (./m < dbs.sql).
11, ./stop_all
12. Add "table_definition_cache = 400" and "innodb_open_files = 400" to master and slave's my.cnf (master/my.sandbox.cnf, node1/my.sandbox.cnf, and node2/my.sandbox.cnf)
13, ./start_all
14, ./m -e "DELETE FROM db444.t; DROP DATABASE db444"
15, If the command in Step 14 blocks, the bug is reproduced on master. If the above command finishes, run "./s1 -e 'show slave status\G' and "./s2 -e 'show slave status\G'". If one slave stucks in "System lock" state, the bug is reproduced on the slave. If both slaves are OK, go back to Step 14 and try a different database (db1, db2, ... db999) until the bug is reproduced.
Possible Explanation:
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever.
Possible Fix:
I fixed my copy of PXC by applying the patch attached here. I am not very familiar with MySQL source code, so I am not sure if my fix is the right way to fix the problem or not. But, I cannot reproduce the problem any more after the fix. |
This bug is not directly related to replication. We found the problem on slaves first, so STRs are based on our production setup which has a master-slave setup.
Steps to Reproduce:
1, wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster-56/LATEST/source/Percona-XtraDB-Cluster-5.6.15.tar.gz
2, tar xzvf Percona-XtraDB-Cluster-5.6.15.tar.gz
3, cd Percona-XtraDB-Cluster-5.6.15
4, cmake .
5, make
6, cd
7, Install mysqlsandbox (http://mysqlsandbox.net/index.html)
8, make_sandbox_from_source ~/Percona-XtraDB-Cluster-5.6.15/ replication
9, Go to the sandbox directory (~/sandboxes/rsandbox_mysql-5_6_15 on my machine).
10, Import the attached sql file into master (./m < dbs.sql).
11, ./stop_all
12. Add "table_definition_cache = 400" and "innodb_open_files = 400" to master and slave's my.cnf (master/my.sandbox.cnf, node1/my.sandbox.cnf, and node2/my.sandbox.cnf)
13, ./start_all
14, ./m -e "DELETE FROM db444.t; DROP DATABASE db444"
15, If the command in Step 14 blocks, the bug is reproduced on master. If the above command finishes, run "./s1 -e 'show slave status\G' and "./s2 -e 'show slave status\G'". If one slave stucks in "System lock" state, the bug is reproduced on the slave. If both slaves are OK, go back to Step 14 and try a different database (db1, db2, ... db999) until the bug is reproduced.
Possible Explanation:
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever.
Possible Fix:
I fixed my copy of PXC by applying the patch attached here. I am not very familiar with MySQL source code, so I am not sure if my fix is the right way to fix the problem or not. But, I cannot reproduce the problem any more after the fix. |
|
| 2014-04-19 02:17:18 |
Raghavendra D Prabhu |
bug task added |
|
percona-xtradb-cluster |
|
| 2014-04-21 21:40:01 |
Chehai Wu |
description |
This bug is not directly related to replication. We found the problem on slaves first, so STRs are based on our production setup which has a master-slave setup.
Steps to Reproduce:
1, wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster-56/LATEST/source/Percona-XtraDB-Cluster-5.6.15.tar.gz
2, tar xzvf Percona-XtraDB-Cluster-5.6.15.tar.gz
3, cd Percona-XtraDB-Cluster-5.6.15
4, cmake .
5, make
6, cd
7, Install mysqlsandbox (http://mysqlsandbox.net/index.html)
8, make_sandbox_from_source ~/Percona-XtraDB-Cluster-5.6.15/ replication
9, Go to the sandbox directory (~/sandboxes/rsandbox_mysql-5_6_15 on my machine).
10, Import the attached sql file into master (./m < dbs.sql).
11, ./stop_all
12. Add "table_definition_cache = 400" and "innodb_open_files = 400" to master and slave's my.cnf (master/my.sandbox.cnf, node1/my.sandbox.cnf, and node2/my.sandbox.cnf)
13, ./start_all
14, ./m -e "DELETE FROM db444.t; DROP DATABASE db444"
15, If the command in Step 14 blocks, the bug is reproduced on master. If the above command finishes, run "./s1 -e 'show slave status\G' and "./s2 -e 'show slave status\G'". If one slave stucks in "System lock" state, the bug is reproduced on the slave. If both slaves are OK, go back to Step 14 and try a different database (db1, db2, ... db999) until the bug is reproduced.
Possible Explanation:
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever.
Possible Fix:
I fixed my copy of PXC by applying the patch attached here. I am not very familiar with MySQL source code, so I am not sure if my fix is the right way to fix the problem or not. But, I cannot reproduce the problem any more after the fix. |
Steps to Reproduce:
1, Install Percona-XtraDB-Cluster-5.6.15-25.5.759
2, mysql < dbs.sql (download it from Bug Attachments)
3, Add "table_definition_cache = 400" and "innodb_open_files = 400" to my.cnf.
4, restart PXC.
5, mysql -e "DELETE FROM db444.t; DROP DATABASE db444"
6, If the command in Step 5 blocks, the bug is reproduced. Otherwise, go back to Step 5 and try a different database (db1, db2, ... db999) until the bug is reproduced.
Possible Explanation:
In storage/innobase/row/row0upd.cc:wsrep_row_upd_check_foreign_constraints, when foreign->referenced_table is NULL, dict_table_open_on_name is called to increase foreign->referenced_table's reference count. However, dict_table_close is never called to decrease the reference count.
This makes some table's reference count non-zero forever. As a result, when MySQL drops a database, it could enter into an infinite loop (see storage/innobase/row/row0mysql.cc:row_drop_database_for_mysql). If this happens in a slave SQL thread, the thread is blocked forever.
Possible Fix:
I fixed my copy of PXC by applying the patch attached here. I am not very familiar with MySQL source code, so I am not sure if my fix is the right way to fix the problem or not. But, I cannot reproduce the problem any more after the fix. |
|
| 2014-04-23 20:35:31 |
Chehai Wu |
attachment added |
|
bug.diff https://bugs.launchpad.net/codership-mysql/+bug/1309241/+attachment/4094687/+files/bug.diff |
|
| 2014-04-23 20:35:55 |
Chehai Wu |
attachment removed |
Possible Fix https://bugs.launchpad.net/codership-mysql/+bug/1309241/+attachment/4087725/+files/my_fix.diff |
|
|
| 2014-05-12 01:58:59 |
Yan Zhang |
codership-mysql: status |
New |
Confirmed |
|
| 2014-05-12 01:59:03 |
Yan Zhang |
codership-mysql: importance |
Undecided |
Medium |
|
| 2014-05-12 02:34:36 |
Launchpad Janitor |
branch linked |
|
lp:codership-mysql |
|
| 2014-05-12 02:45:10 |
Yan Zhang |
codership-mysql: assignee |
|
Yan Zhang (yan.zhang) |
|
| 2014-05-12 03:08:04 |
Raghavendra D Prabhu |
nominated for series |
|
percona-xtradb-cluster/5.6 |
|
| 2014-05-12 03:08:04 |
Raghavendra D Prabhu |
bug task added |
|
percona-xtradb-cluster/5.6 |
|
| 2014-05-12 03:08:04 |
Raghavendra D Prabhu |
nominated for series |
|
percona-xtradb-cluster/trunk |
|
| 2014-05-12 03:08:04 |
Raghavendra D Prabhu |
bug task added |
|
percona-xtradb-cluster/trunk |
|
| 2014-05-12 03:08:33 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.6: milestone |
|
5.6.17-25.6 |
|
| 2014-05-12 03:08:35 |
Raghavendra D Prabhu |
percona-xtradb-cluster/trunk: milestone |
|
future-5.5 |
|
| 2014-05-12 09:11:57 |
Yan Zhang |
nominated for series |
|
codership-mysql/5.5 |
|
| 2014-05-12 09:11:57 |
Yan Zhang |
bug task added |
|
codership-mysql/5.5 |
|
| 2014-05-12 09:11:57 |
Yan Zhang |
nominated for series |
|
codership-mysql/5.6 |
|
| 2014-05-12 09:11:57 |
Yan Zhang |
bug task added |
|
codership-mysql/5.6 |
|
| 2014-05-12 09:12:13 |
Yan Zhang |
codership-mysql/5.5: status |
New |
Invalid |
|
| 2014-05-12 09:12:23 |
Yan Zhang |
codership-mysql/5.5: assignee |
|
Yan Zhang (yan.zhang) |
|
| 2014-05-12 09:12:34 |
Yan Zhang |
codership-mysql/5.6: milestone |
|
5.6.17-25.6 |
|
| 2014-06-06 14:48:31 |
Yan Zhang |
codership-mysql/5.6: status |
Confirmed |
Fix Committed |
|
| 2014-06-06 14:49:39 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.6: status |
New |
Fix Committed |
|
| 2014-07-02 16:23:30 |
Raghavendra D Prabhu |
percona-xtradb-cluster/trunk: status |
New |
Invalid |
|
| 2014-07-02 16:23:33 |
Raghavendra D Prabhu |
percona-xtradb-cluster/trunk: milestone |
future-5.5 |
|
|
| 2014-07-22 06:52:27 |
Hrvoje Matijakovic |
percona-xtradb-cluster/5.6: status |
Fix Committed |
Fix Released |
|
| 2015-02-09 23:00:39 |
Alex Yurchenko |
codership-mysql/5.6: status |
Fix Committed |
Fix Released |
|
| 2015-02-09 23:00:46 |
Alex Yurchenko |
codership-mysql/5.6: milestone |
5.6.20-25.6 |
5.6.21-25.9 |
|
