Odd are... bug in drizzleslap. I've not looked at it in sometime. It
is still using the my_getopt junk so it links to the mysys mess... so
it probably has stuff dinking with its threads.
Cheers,
-Brian
On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
> Something seems to be broken once we exceed the back_log count. Once
> getting drizzleslap to hang, I dumped the active sockets for each
> process:
>
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzlesl 32531 eday 3u IPv4 10320113 TCP localhost:54858-
> >localhost:9306 (CLOSE_WAIT)
> drizzlesl 32531 eday 185u IPv4 10324113 TCP localhost:56240-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 186u IPv4 10324115 TCP localhost:56241-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 187u IPv4 10324117 TCP localhost:56242-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 188u IPv4 10324119 TCP localhost:56243-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 189u IPv4 10324121 TCP localhost:56244-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 190u IPv4 10324123 TCP localhost:56245-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 191u IPv4 10324125 TCP localhost:56246-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 192u IPv4 10324127 TCP localhost:56247-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 193u IPv4 10324129 TCP localhost:56248-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 194u IPv4 10324131 TCP localhost:56249-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 195u IPv4 10324133 TCP localhost:56250-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 196u IPv4 10324135 TCP localhost:56251-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 197u IPv4 10324137 TCP localhost:56252-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 198u IPv4 10324139 TCP localhost:56253-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 199u IPv4 10324141 TCP localhost:56254-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 200u IPv4 10324143 TCP localhost:56255-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 201u IPv4 10324145 TCP localhost:56256-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 202u IPv4 10324147 TCP localhost:56257-
> >localhost:9306 (ESTABLISHED)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzled 20886 eday 10u IPv4 10302592 TCP *:9306 (LISTEN)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> tcp 0 0 127.0.0.1:56243 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56255 127.0.0.1:9306
> ESTABLISHED
> tcp 1 0 127.0.0.1:54858 127.0.0.1:9306
> CLOSE_WAIT
> tcp 0 0 127.0.0.1:56247 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56252 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56249 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56256 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56251 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56240 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56257 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56254 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56245 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56241 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56246 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56242 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56248 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56250 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56244 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56253 127.0.0.1:9306
> ESTABLISHED
>
>
> As you can see, drizzleslap thinks it has valid connections and are
> all blocking in poll(), but the server has no knowledge of the
> active connections. The OS should be causing poll() to return
> POLLERR or POLLHUP inside of drizzleslap, rather than have them
> block indefinitely for POLLIN.
>
> --
> Drizzleslap hangs with over 175 connections
> https://bugs.launchpad.net/bugs/455137
> You received this bug notification because you are a member of
> Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
>
> Bug description:
> Running drizzleslap with any number connections great than 175
> causes it to hang:
>
> Example:
> drizzleslap --port=9306 --label=write --concurrency=200 --
> iterations=10 --engine=innodb --number-int
> -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-
> add-autoincrement --auto-genera
> te-sql-load-type=write --number-of-queries=100
>
> attaching to the process through gdb gives:
> (gdb) bt
> #0 0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 ()
> from /lib/libpthread.so.0
> #1 0x00000000004051c2 in run_scheduler (sptr=0xa462e0,
> stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> #2 0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80,
> current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> #3 0x0000000000409101 in main (argc=0, argv=0xa44178) at client/
> drizzleslap.cc:377
>
> adding -v -v -v for lots of verbose output shows that it hangs while
> doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL,
> 1576223060,942232734
> ,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH
> ','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l
> ','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW
> ');
> INSERT INTO t1 VALUES (NULL,
> 1322166549,823399162
> ,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg
> ','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L
> ','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF
> ');
> INSERT INTO t1 VALUES (NULL,
> 116314197,805287600
> ,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4
> ','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3
> ','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn
> ');
> INSERT INTO t1 VALUES (NULL,
> 1407841593,32683953
> ,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880
> ','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH
> ','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM
> ');
> INSERT INTO t1 VALUES (NULL,
> 1471079933,266531426
> ,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc
> ','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn
> ','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O
> ');
>
> For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/
> to only use a max of 175 connections
Hi!
Odd are... bug in drizzleslap. I've not looked at it in sometime. It
is still using the my_getopt junk so it links to the mysys mess... so
it probably has stuff dinking with its threads.
Cheers,
-Brian
On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
> Something seems to be broken once we exceed the back_log count. Once 1:~/bzr/ drizzle/ eday-dev/ tests$ lsof -p 32531 -i4 -a 1:~/bzr/ drizzle/ eday-dev/ tests$ lsof -p 20886 -i4 -a 1:~/bzr/ drizzle/ eday-dev/ tests$ netstat -n|grep 9306 /bugs.launchpad .net/bugs/ 455137 char-cols= 3 --auto-generate-sql --auto- generate- sql- load-type= write --number- of-queries= 100 cond_timedwait@ @GLIBC_ 2.3.2 () .so.0 drizzleslap. cc:2137 be80, drizzleslap. cc:495 942232734 B0cewKe8d5AyZL6 EZ01qmkDAILKLh8 9t8lXwx9xQ67OXK WltWek9EpBYAXlj fvcfY0cM1Zo8YDX sbHlynL7uAIQCi4 wDrOtFOTdQuQQQX pI6y3wmH 219NeRHwlbRatQZ pMJ6IrPYppKkHTT 2OWBf1WpxFOHpAb MzpZx7OEX64rqDD BFdaIGbo5O3MhlY G7XyZK5xQTvWKlc Zxj2X2JyiGMee4N ACMZ2vl8l gccMlq0FmNIrmjt hL1IPWzrCHuRYgk 2KDWaqKb57qNycz rAKlSl7JcJcMe8u N26SYhItGF0yXyb WhdqQxRQHvltIGu CfwJzTs9M0OEyv4 PHl274sQW 823399162 CnJQmEMBO76N3yq ZzCZtkwaSrBGc93 XDjqfLwtOK0WkwK aLJeu5MrfiaiyAj uZwdNRyKKMYClG8 5iMHKGMaztHPR7S A11yWPPKddj9pKQ qH8CQLKg cOHw1uXAkjSQ1Te Jqy0lAPqoQ2lKzs LR8WG9PnC1NYRG1 xbrvbdy1ReJLNfC 8sgGFwIgbMhyu0E wxinsHGiAbKLoct 1klhs0d2Y4O53J5 972run13L oWhKaxQTGNfFIMA 5F5QJj0k2A2WwxN Rlt8yS5GpEfSTxj XuycD9CDtEGnd4d 0XEufCEdS3j0O4p 7ckx4x7G2sNAg0F H0mXmIc0nwwWoKa eJ2E873FF 6T6SHI3vdx1lw4t C2liAH5PySQ2dej fcCd5Lv0HOxA2uF nyHYYJhdcf52ZbY GE9KkEifErw31qx 8dnH1KSax4DvzmW XvcY7KmhssOowIW lQzZc0l4 CoilYbACNSi0mhP MZARzlcWEcg6zkQ 3o6eafDcrS4QSq7 u5Z4YYh0vwSbuRB KN0QtQ562pQyFmg EYlnT9CA176Ta0p lC4dfxvkvpZDngz KvY5it7m3 CMMzBoCrSBBFb8y SFkYGWQmhKxeRC5 5ORSNWGbLceYsEh iaoIy4gOiMlfs4r xA7GeMnmp0ESZ6a 6g8mz7jXNPt85mD wbN4ktIXf8Xlcwk AuOAZF9Gn PpG3P5nRrBWOJXY y46WWkhG3JAQhw8 QEpaH7f4QN7wfIS CGOInuuK3qX5G41 GM76wG5SCXrKKMW gan4kQrXmmL9rTD kmbskN8prLCaZYx 3CNiH880 bz6GiJTHk16n6TM FdWF8HtsCM6DRxo PZfNhEiAXQ4T5Ay RPSyYS7PLJfJO6G 5XpkuXycZLuTqsf hBw91Sd8lqSO9sX Qx27apWOHweCPKw 517t29EgH WS2gOgsp0gmunYw wmSEdrrtFDg5PE2 2cM4Gek0Kdg6XSW JGGoWuXEneJKjcQ lf20jBeKBypkxK4 RXKkBrWax9CGTvl vZOqZz13BC20orC sIcDunwGM 266531426 xe9yR02BJeO5IKM g5o67bKlI7gjJQH bNokD7lGIwmjt4v Fc1vb0Xvdh3HPEb ipzMQCT3Koz7Ns3 I0gJv7JQ2x8x6P4 i7tZMBlrEyXDxuy sDqOOlXc qOGaXE5k30av5OK T2r56HW2ZuXszrx eiEm8lsDLv5oiA4 2g6md4v86K2T5tL uNv893l18zoEfNO jPwq3amratBSE8D b3dM3eHhYQgKgJj wu1lkwJXn aWn6oJIKWuGOnt1 jCOXpYMu2LDsG85 EaRtYFCrbadjyOd r7hFWXfkHiyWQEX NKxqDW5HDYrq8h6 dP5AYcZCKrmADCG kccaQp4O69opzoN 5PoASCe1O hudson. drizzle. org/view/ Drizzle- staging/ job/Drizzleslap -staging/
> getting drizzleslap to hang, I dumped the active sockets for each
> process:
>
> eday@orisndriz0
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzlesl 32531 eday 3u IPv4 10320113 TCP localhost:54858-
> >localhost:9306 (CLOSE_WAIT)
> drizzlesl 32531 eday 185u IPv4 10324113 TCP localhost:56240-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 186u IPv4 10324115 TCP localhost:56241-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 187u IPv4 10324117 TCP localhost:56242-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 188u IPv4 10324119 TCP localhost:56243-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 189u IPv4 10324121 TCP localhost:56244-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 190u IPv4 10324123 TCP localhost:56245-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 191u IPv4 10324125 TCP localhost:56246-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 192u IPv4 10324127 TCP localhost:56247-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 193u IPv4 10324129 TCP localhost:56248-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 194u IPv4 10324131 TCP localhost:56249-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 195u IPv4 10324133 TCP localhost:56250-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 196u IPv4 10324135 TCP localhost:56251-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 197u IPv4 10324137 TCP localhost:56252-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 198u IPv4 10324139 TCP localhost:56253-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 199u IPv4 10324141 TCP localhost:56254-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 200u IPv4 10324143 TCP localhost:56255-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 201u IPv4 10324145 TCP localhost:56256-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 202u IPv4 10324147 TCP localhost:56257-
> >localhost:9306 (ESTABLISHED)
> eday@orisndriz0
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzled 20886 eday 10u IPv4 10302592 TCP *:9306 (LISTEN)
> eday@orisndriz0
> tcp 0 0 127.0.0.1:56243 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56255 127.0.0.1:9306
> ESTABLISHED
> tcp 1 0 127.0.0.1:54858 127.0.0.1:9306
> CLOSE_WAIT
> tcp 0 0 127.0.0.1:56247 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56252 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56249 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56256 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56251 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56240 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56257 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56254 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56245 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56241 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56246 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56242 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56248 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56250 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56244 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56253 127.0.0.1:9306
> ESTABLISHED
>
>
> As you can see, drizzleslap thinks it has valid connections and are
> all blocking in poll(), but the server has no knowledge of the
> active connections. The OS should be causing poll() to return
> POLLERR or POLLHUP inside of drizzleslap, rather than have them
> block indefinitely for POLLIN.
>
> --
> Drizzleslap hangs with over 175 connections
> https:/
> You received this bug notification because you are a member of
> Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
>
> Bug description:
> Running drizzleslap with any number connections great than 175
> causes it to hang:
>
> Example:
> drizzleslap --port=9306 --label=write --concurrency=200 --
> iterations=10 --engine=innodb --number-int
> -cols=2 --number-
> add-autoincrement --auto-genera
> te-sql-
>
> attaching to the process through gdb gives:
> (gdb) bt
> #0 0x00007f3c0d98756d in pthread_
> from /lib/libpthread
> #1 0x00000000004051c2 in run_scheduler (sptr=0xa462e0,
> stmts=0xa44350, concur=200, limit=0) at client/
> #2 0x0000000000408e12 in concurrency_loop (con=0x7fff16fd
> current=200, eptr=0xa45970) at client/
> #3 0x0000000000409101 in main (argc=0, argv=0xa44178) at client/
> drizzleslap.cc:377
>
> adding -v -v -v for lots of verbose output shows that it hangs while
> doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL,
> 1576223060,
> ,'nHujWOi8H3Sww
> ','3gREyYw2S9ls
> ','4dgQpQ2FFmHe
> ');
> INSERT INTO t1 VALUES (NULL,
> 1322166549,
> ,'7ugNokudvtyn2
> ','vKZ4d0C52570
> ','3eiPRAISuv1e
> ');
> INSERT INTO t1 VALUES (NULL,
> 116314197,805287600
> ,'e8YWNpN1C9JgA
> ','Qi9fT8HrX4S4
> ','8t2WJuJx7Fam
> ');
> INSERT INTO t1 VALUES (NULL,
> 1407841593,32683953
> ,'YZeAmRm1NWuht
> ','XumsvTL8Dzbx
> ','zMdGYMnyJhxl
> ');
> INSERT INTO t1 VALUES (NULL,
> 1471079933,
> ,'a1vMmjn2ZKuaF
> ','4QZ4NXTJ5SDe
> ','lGr5WzIy2vC6
> ');
>
> For now modifying all tests at http://
> to only use a max of 175 connections