Drizzle transaction log showing differences from server state in concurrent testing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Drizzle |
Fix Released
|
Critical
|
David Shrewsbury | ||
7.0 |
Fix Released
|
Critical
|
David Shrewsbury |
Bug Description
When testing Drizzle's transaction log in a concurrent environment with the randgen, we are seeing differences between the server's state and the SQL captured in the transaction log. The failures appear to be the results of INSERT's, UPDATE's, and DELETE's.
The test consists of executing several queries (in several threads) against the server. We then generate SQL from the transaction log and use the SQL to populate a validation server. We compare Drizzledump output between 'master' and 'slave' to validate.
The failure is generally consistent, but appears to be subject to timing issues. The randgen fork()'s off processes for each 'thread' used and the exact timing and interaction between threads varies. However, we see failures at certain thread/query_count combinations:
2 threads, 2500 queries
5 threads, 70-75 queries
10 thread, 50 queries.
This test grammar was designed to produce a higher percentage of valid queries and tables can easily grow quite large.
At present, I am not certain what is causing the issue, but it is an interaction between connections. These failures are not repeatable in single-thread scenarios.
TO REPEAT:
1) Start a drizzle server to test, enable the transaction log
./test-run --start-and-exit --drizzled=
2) Start a second server to validate (expected port 9307)
3) Run the randgen and observe the results. Command line:
perl ./gentest.pl \
--dsn=dbi:
--gendata=
--queries=70 --threads=5 --sqltrace --debug --Reporter=
Sample output:
# 2010-11-09T11:51:18 Executing diff --unified /tmp//translog_
--- /tmp//translog_
+++ /tmp//translog_
@@ -1014,21 +1014,46 @@
INSERT INTO `a` VALUES (NULL,'
INSERT INTO `a` VALUES (NULL,'
INSERT INTO `a` VALUES (NULL,'
+INSERT INTO `a` VALUES ('-732679364377
+INSERT INTO `a` VALUES ('-696819452344
INSERT INTO `a` VALUES (NULL,'
+INSERT INTO `a` VALUES ('-696819452344
+INSERT INTO `a` VALUES ('-688206318057
Related branches
- Drizzle Developers: Pending requested
-
Diff: 411 lines (+338/-7)6 files modifieddrizzled/statement/rollback_to_savepoint.cc (+18/-3)
drizzled/statement/savepoint.cc (+19/-3)
plugin/transaction_log/tests/r/savepoint.result (+199/-0)
plugin/transaction_log/tests/t/savepoint-master.opt (+1/-0)
plugin/transaction_log/tests/t/savepoint.test (+100/-0)
plugin/transaction_log/tests/t/temp_tables.test (+1/-1)
Changed in drizzle: | |
assignee: | nobody → David Shrewsbury (dshrews) |
status: | New → Confirmed |
tags: | added: replication |
Changed in drizzle: | |
status: | Confirmed → In Progress |
importance: | Undecided → Critical |
description: | updated |
Tarball with:
* transaction log after the randgen run
* dumpfile to create the initial test bed
* dumpfile from master post-run
* dumpfile from slave post-run (generated / populated from transaction log via transaction_reader)
* output from the randgen - this is the list of queries generated and sent to the server. Each query includes the thread_id in the comments.
This is from a 5 thread, 75 query run. The diff is easier than in many previous runs (only one query appears to differ between master and slave): rpl_dump_ 6637_9306. sql 2010-11-09 12:20:38.670957495 -0500 rpl_dump_ 6637_9307. sql 2010-11-09 12:20:39.171210764 -0500 ,'d','' ,270270464, 64290816, 9,'EOADF' ,'e','2' ,'KQZTC' ,'VWWRE' ,'d','' ,990,'my' ,'say', 'just', '7','k' ,'3','' ,'y',5, '5'); ,'o','' ,5,6,NULL, 'EPLEV' ,'m','6' ,'f','not' ,'w','' ,991,'RHVXU' ,'s','I\ 'm','9' ,'f','- 758406177249191 5264',' ','j',186358169 6,'9'); ,'x','' ,2,166133760, NULL,'RQWBU' ,'g','3' ,'your' ,'QTFXV' ,'h','' ,992,'a' ,'hey', 'going' ,'2','p' ,'5483695496277 000192' ,'','e' ,751239168, '57691111226616 05376') ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',993, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',994, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',995, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',996, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',997, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',998, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',999, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1000, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1001, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1002, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1003, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1004, 'NILAN' ,'about' ,'CVZCR' ,'5','z' ,'-430741156860 3168768' ,'','I' ,9,NULL) ; ,'e','' ,3,5,NULL, 'a','z' ,NULL,' OIILA', 'r','o' ,'',1005, 'NILA.. .
--- /tmp//translog_
+++ /tmp//translog_
@@ -1930,6 +1930,22 @@
INSERT INTO `aa` VALUES ('','e'
INSERT INTO `aa` VALUES ('','z'
INSERT INTO `aa` VALUES ('','o'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'
+INSERT INTO `aa` VALUES ('','been'