Bug #455137 “Drizzleslap hangs with over 175 connections” : Bugs : Drizzle

Lee Bieber (kalebral-deactivatedaccount) on 2009-10-19

Changed in drizzle:
importance:	Undecided → Critical
milestone:	none → bell
status:	New → Confirmed
assignee:	nobody → Eric Day (eday)

Revision history for this message

fmpwizard (diego-fmpwizard) wrote on 2009-10-19:

#1

This may be similar to this bug:
http://bugs.mysql.com/bug.php?id=43278
The mysql proxy used to hang when running against mysqlslap (and also against drizzleslap using the mysql compatible mode)

The fix there was
http://bazaar.launchpad.net/~mysql-proxy-developers/mysql-proxy/trunk/revision/708

Basically increasing
listen(con->fd, 8) to
listen(con->fd, 128)

But looking at the drizzle code, that value is not hardcoded:

I found in drizzled/plugin/listen_tcp.cc
this line:

if (listen(fd, (int) back_log) < 0)

and I found that "back_log" is a parameter you can pass to drizzled on startup (or in drizzle.cnf) as --back_log

$ ./sbin/drizzled --help | grep -i back_log
--back_log=# The number of outstanding connection requests Drizzle can
back_log 50

Lee: Could you see if http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ is passing any value for back_log?

Thanks

Diego

Revision history for this message

Lee Bieber (kalebral-deactivatedaccount) wrote on 2009-10-19:

#2

We are not passing any value for back_log when starting drizzled

Revision history for this message

Lee Bieber (kalebral-deactivatedaccount) wrote on 2009-10-19:

#3

If do start drizzled with --back_log=500 then I am able to run tests with higher concurrency

Revision history for this message

Eric Day (eday) wrote on 2009-11-06:

#4

Download full text (4.1 KiB)

Something seems to be broken once we exceed the back_log count. Once getting drizzleslap to hang, I dumped the active sockets for each process:

eday@orisndriz01:~/bzr/1:~/bzr/1:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
DEVICE SIZE NODE NAME
TCP localhost:54858->localhost:9306 (CLOSE_WAIT)
TCP localhost:56240->localhost:9306 (ESTABLISHED)
TCP localhost:56241->localhost:9306 (ESTABLISHED)
TCP localhost:56242->localhost:9306 (ESTABLISHED)
TCP localhost:56243->localhost:9306 (ESTABLISHED)
TCP localhost:56244->localhost:9306 (ESTABLISHED)
TCP localhost:56245->localhost:9306 (ESTABLISHED)
TCP localhost:56246->localhost:9306 (ESTABLISHED)
TCP localhost:56247->localhost:9306 (ESTABLISHED)
TCP localhost:56248->localhost:9306 (ESTABLISHED)
TCP localhost:56249->localhost:9306 (ESTABLISHED)
TCP localhost:56250->localhost:9306 (ESTABLISHED)
TCP localhost:56251->localhost:9306 (ESTABLISHED)
TCP localhost:56252->localhost:9306 (ESTABLISHED)
TCP localhost:56253->localhost:9306 (ESTABLISHED)
TCP localhost:56254->localhost:9306 (ESTABLISHED)
TCP localhost:56255->localhost:9306 (ESTABLISHED)
TCP localhost:56256->localhost:9306 (ESTABLISHED)
TCP localhost:56257->localhost:9306 (ESTABLISHED)
/>drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
DEVICE SIZE NODE NAME
TCP *:9306 (LISTEN)
/>drizzle/eday-dev/tests$ netstat -n|grep 9306
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 CLOSE_WAIT
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED
127.0.0.1:9306 ESTABLISHED

Hi!

Odd are... bug in drizzleslap. I've not looked at it in sometime. It  
is still using the my_getopt junk so it links to the mysys mess... so  
it probably has stuff dinking with its threads.

Cheers,
	-Brian

On Nov 5, 2009, at 6:33 PM, Eric Day wrote:

> Something seems to be broken once we exceed the back_log count. Once
> getting drizzleslap to hang, I dumped the active sockets for each
> process:
>
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> COMMAND     PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> drizzlesl 32531 eday    3u  IPv4 10320113       TCP localhost:54858- 
> >localhost:9306 (CLOSE_WAIT)
> drizzlesl 32531 eday  185u  IPv4 10324113       TCP localhost:56240- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  186u  IPv4 10324115       TCP localhost:56241- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  187u  IPv4 10324117       TCP localhost:56242- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  188u  IPv4 10324119       TCP localhost:56243- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  189u  IPv4 10324121       TCP localhost:56244- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  190u  IPv4 10324123       TCP localhost:56245- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  191u  IPv4 10324125       TCP localhost:56246- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  192u  IPv4 10324127       TCP localhost:56247- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  193u  IPv4 10324129       TCP localhost:56248- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  194u  IPv4 10324131       TCP localhost:56249- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  195u  IPv4 10324133       TCP localhost:56250- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  196u  IPv4 10324135       TCP localhost:56251- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  197u  IPv4 10324137       TCP localhost:56252- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  198u  IPv4 10324139       TCP localhost:56253- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  199u  IPv4 10324141       TCP localhost:56254- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  200u  IPv4 10324143       TCP localhost:56255- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  201u  IPv4 10324145       TCP localhost:56256- 
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday  202u  IPv4 10324147       TCP localhost:56257- 
> >localhost:9306 (ESTABLISHED)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> COMMAND    PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> drizzled 20886 eday   10u  IPv4 10302592       TCP *:9306 (LISTEN)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> tcp        0      0 127.0.0.1:56243         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56255         127.0.0.1:9306           
> ESTABLISHED
> tcp        1      0 127.0.0.1:54858         127.0.0.1:9306           
> CLOSE_WAIT
> tcp        0      0 127.0.0.1:56247         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56252         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56249         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56256         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56251         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56240         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56257         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56254         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56245         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56241         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56246         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56242         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56248         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56250         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56244         127.0.0.1:9306           
> ESTABLISHED
> tcp        0      0 127.0.0.1:56253         127.0.0.1:9306           
> ESTABLISHED
>
>
> As you can see, drizzleslap thinks it has valid connections and are  
> all blocking in poll(), but the server has no knowledge of the  
> active connections. The OS should be causing poll() to return  
> POLLERR or POLLHUP inside of drizzleslap, rather than have them  
> block indefinitely for POLLIN.
>
> -- 
> Drizzleslap hangs with over 175 connections
> https://bugs.launchpad.net/bugs/455137
> You received this bug notification because you are a member of  
> Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
>
> Bug description:
> Running drizzleslap with any number connections great than 175  
> causes it to hang:
>
> Example:
> drizzleslap   --port=9306 --label=write --concurrency=200 -- 
> iterations=10 --engine=innodb --number-int
> -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql- 
> add-autoincrement --auto-genera
> te-sql-load-type=write --number-of-queries=100
>
> attaching to the process through gdb gives:
> (gdb) bt
> #0  0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 ()  
> from /lib/libpthread.so.0
> #1  0x00000000004051c2 in run_scheduler (sptr=0xa462e0,  
> stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> #2  0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80,  
> current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> #3  0x0000000000409101 in main (argc=0, argv=0xa44178) at client/ 
> drizzleslap.cc:377
>
> adding -v -v -v for lots of verbose output shows that it hangs while  
> doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL, 
> 1576223060,942232734 
> ,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH 
> ','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l 
> ','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW 
> ');
> INSERT INTO t1 VALUES (NULL, 
> 1322166549,823399162 
> ,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg 
> ','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L 
> ','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF 
> ');
> INSERT INTO t1 VALUES (NULL, 
> 116314197,805287600 
> ,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4 
> ','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3 
> ','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn 
> ');
> INSERT INTO t1 VALUES (NULL, 
> 1407841593,32683953 
> ,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880 
> ','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH 
> ','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM 
> ');
> INSERT INTO t1 VALUES (NULL, 
> 1471079933,266531426 
> ,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc 
> ','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn 
> ','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O 
> ');
>
> For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ 
>  to only use a max of 175 connections

Revision history for this message

Eric Day (eday) wrote on 2009-11-06:

#6

Download full text (12.5 KiB)

Hmm, I don't think this is even mysys's fault, but I could be
wrong. The poll() is down in libdrizzle, and normally if a connection
fails poll() will just return. I'm actually leaning more towards a
OS bug with local connections with backlog overruns, but I'd have to
create a quick test program to verify.

But who knows, mysys has surprised me before. :)

-Eric

On Fri, Nov 06, 2009 at 02:56:22AM -0000, Brian Aker wrote:
> Hi!
>
> Odd are... bug in drizzleslap. I've not looked at it in sometime. It
> is still using the my_getopt junk so it links to the mysys mess... so
> it probably has stuff dinking with its threads.
>
> Cheers,
> -Brian
>
> On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
>
> > Something seems to be broken once we exceed the back_log count. Once
> > getting drizzleslap to hang, I dumped the active sockets for each
> > process:
> >
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> > COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> > drizzlesl 32531 eday 3u IPv4 10320113 TCP localhost:54858-
> > >localhost:9306 (CLOSE_WAIT)
> > drizzlesl 32531 eday 185u IPv4 10324113 TCP localhost:56240-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 186u IPv4 10324115 TCP localhost:56241-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 187u IPv4 10324117 TCP localhost:56242-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 188u IPv4 10324119 TCP localhost:56243-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 189u IPv4 10324121 TCP localhost:56244-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 190u IPv4 10324123 TCP localhost:56245-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 191u IPv4 10324125 TCP localhost:56246-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 192u IPv4 10324127 TCP localhost:56247-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 193u IPv4 10324129 TCP localhost:56248-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 194u IPv4 10324131 TCP localhost:56249-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 195u IPv4 10324133 TCP localhost:56250-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 196u IPv4 10324135 TCP localhost:56251-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 197u IPv4 10324137 TCP localhost:56252-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 198u IPv4 10324139 TCP localhost:56253-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 199u IPv4 10324141 TCP localhost:56254-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 200u IPv4 10324143 TCP localhost:56255-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 201u IPv4 10324145 TCP localhost:56256-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 202u IPv4 10324147 TCP localhost:56257-
> > >localhost:9306 (ESTABLISHED)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> > COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> > drizzled 20886...

Hmm, I don't think this is even mysys's fault, but I could be
wrong. The poll() is down in libdrizzle, and normally if a connection
fails poll() will just return. I'm actually leaning more towards a
OS bug with local connections with backlog overruns, but I'd have to
create a quick test program to verify.

But who knows, mysys has surprised me before. :)

-Eric

On Fri, Nov 06, 2009 at 02:56:22AM -0000, Brian Aker wrote:
> Hi!
> 
> Odd are... bug in drizzleslap. I've not looked at it in sometime. It  
> is still using the my_getopt junk so it links to the mysys mess... so  
> it probably has stuff dinking with its threads.
> 
> Cheers,
> 	-Brian
> 
> On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
> 
> > Something seems to be broken once we exceed the back_log count. Once
> > getting drizzleslap to hang, I dumped the active sockets for each
> > process:
> >
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> > COMMAND     PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> > drizzlesl 32531 eday    3u  IPv4 10320113       TCP localhost:54858- 
> > >localhost:9306 (CLOSE_WAIT)
> > drizzlesl 32531 eday  185u  IPv4 10324113       TCP localhost:56240- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  186u  IPv4 10324115       TCP localhost:56241- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  187u  IPv4 10324117       TCP localhost:56242- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  188u  IPv4 10324119       TCP localhost:56243- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  189u  IPv4 10324121       TCP localhost:56244- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  190u  IPv4 10324123       TCP localhost:56245- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  191u  IPv4 10324125       TCP localhost:56246- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  192u  IPv4 10324127       TCP localhost:56247- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  193u  IPv4 10324129       TCP localhost:56248- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  194u  IPv4 10324131       TCP localhost:56249- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  195u  IPv4 10324133       TCP localhost:56250- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  196u  IPv4 10324135       TCP localhost:56251- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  197u  IPv4 10324137       TCP localhost:56252- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  198u  IPv4 10324139       TCP localhost:56253- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  199u  IPv4 10324141       TCP localhost:56254- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  200u  IPv4 10324143       TCP localhost:56255- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  201u  IPv4 10324145       TCP localhost:56256- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  202u  IPv4 10324147       TCP localhost:56257- 
> > >localhost:9306 (ESTABLISHED)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> > COMMAND    PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> > drizzled 20886 eday   10u  IPv4 10302592       TCP *:9306 (LISTEN)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> > tcp        0      0 127.0.0.1:56243         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56255         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        1      0 127.0.0.1:54858         127.0.0.1:9306           
> > CLOSE_WAIT
> > tcp        0      0 127.0.0.1:56247         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56252         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56249         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56256         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56251         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56240         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56257         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56254         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56245         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56241         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56246         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56242         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56248         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56250         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56244         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56253         127.0.0.1:9306           
> > ESTABLISHED
> >
> >
> > As you can see, drizzleslap thinks it has valid connections and are  
> > all blocking in poll(), but the server has no knowledge of the  
> > active connections. The OS should be causing poll() to return  
> > POLLERR or POLLHUP inside of drizzleslap, rather than have them  
> > block indefinitely for POLLIN.
> >
> > -- 
> > Drizzleslap hangs with over 175 connections
> > https://bugs.launchpad.net/bugs/455137
> > You received this bug notification because you are a member of  
> > Drizzle-
> > developers, which is subscribed to Drizzle.
> >
> > Status in A Lightweight SQL Database for Cloud and Web: Confirmed
> >
> > Bug description:
> > Running drizzleslap with any number connections great than 175  
> > causes it to hang:
> >
> > Example:
> > drizzleslap   --port=9306 --label=write --concurrency=200 -- 
> > iterations=10 --engine=innodb --number-int
> > -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql- 
> > add-autoincrement --auto-genera
> > te-sql-load-type=write --number-of-queries=100
> >
> > attaching to the process through gdb gives:
> > (gdb) bt
> > #0  0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 ()  
> > from /lib/libpthread.so.0
> > #1  0x00000000004051c2 in run_scheduler (sptr=0xa462e0,  
> > stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> > #2  0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80,  
> > current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> > #3  0x0000000000409101 in main (argc=0, argv=0xa44178) at client/ 
> > drizzleslap.cc:377
> >
> > adding -v -v -v for lots of verbose output shows that it hangs while  
> > doing a bunch of inserts:
> > ........
> > INSERT INTO t1 VALUES (NULL, 
> > 1576223060,942232734 
> > ,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH 
> > ','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l 
> > ','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1322166549,823399162 
> > ,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg 
> > ','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L 
> > ','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 116314197,805287600 
> > ,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4 
> > ','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3 
> > ','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1407841593,32683953 
> > ,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880 
> > ','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH 
> > ','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1471079933,266531426 
> > ,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc 
> > ','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn 
> > ','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O 
> > ');
> >
> > For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ 
> >  to only use a max of 175 connections
> 
> -- 
> Drizzleslap hangs with over 175 connections
> https://bugs.launchpad.net/bugs/455137
> You received this bug notification because you are a bug assignee.
> 
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
> 
> Bug description:
> Running drizzleslap with any number connections great than 175 causes it to hang:
> 
> Example:
> drizzleslap   --port=9306 --label=write --concurrency=200 --iterations=10 --engine=innodb --number-int
> -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-genera
> te-sql-load-type=write --number-of-queries=100
> 
> attaching to the process through gdb gives:
> (gdb) bt
> #0  0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> #1  0x00000000004051c2 in run_scheduler (sptr=0xa462e0, stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> #2  0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80, current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> #3  0x0000000000409101 in main (argc=0, argv=0xa44178) at client/drizzleslap.cc:377
> 
> adding -v -v -v for lots of verbose output shows that it hangs while doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL,1576223060,942232734,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW');
> INSERT INTO t1 VALUES (NULL,1322166549,823399162,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF');
> INSERT INTO t1 VALUES (NULL,116314197,805287600,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn');
> INSERT INTO t1 VALUES (NULL,1407841593,32683953,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM');
> INSERT INTO t1 VALUES (NULL,1471079933,266531426,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O');
> 
> For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ to only use a max of 175 connections

Revision history for this message

Eric Day (eday) wrote on 2009-11-06:

#7

Download full text (12.2 KiB)

Also, FWIW, I can't reproduce this on Solaris, seems to just be Linux.

-Eric

On Fri, Nov 06, 2009 at 02:56:22AM -0000, Brian Aker wrote:
> Hi!
>
> Odd are... bug in drizzleslap. I've not looked at it in sometime. It
> is still using the my_getopt junk so it links to the mysys mess... so
> it probably has stuff dinking with its threads.
>
> Cheers,
> -Brian
>
> On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
>
> > Something seems to be broken once we exceed the back_log count. Once
> > getting drizzleslap to hang, I dumped the active sockets for each
> > process:
> >
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> > COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> > drizzlesl 32531 eday 3u IPv4 10320113 TCP localhost:54858-
> > >localhost:9306 (CLOSE_WAIT)
> > drizzlesl 32531 eday 185u IPv4 10324113 TCP localhost:56240-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 186u IPv4 10324115 TCP localhost:56241-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 187u IPv4 10324117 TCP localhost:56242-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 188u IPv4 10324119 TCP localhost:56243-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 189u IPv4 10324121 TCP localhost:56244-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 190u IPv4 10324123 TCP localhost:56245-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 191u IPv4 10324125 TCP localhost:56246-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 192u IPv4 10324127 TCP localhost:56247-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 193u IPv4 10324129 TCP localhost:56248-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 194u IPv4 10324131 TCP localhost:56249-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 195u IPv4 10324133 TCP localhost:56250-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 196u IPv4 10324135 TCP localhost:56251-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 197u IPv4 10324137 TCP localhost:56252-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 198u IPv4 10324139 TCP localhost:56253-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 199u IPv4 10324141 TCP localhost:56254-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 200u IPv4 10324143 TCP localhost:56255-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 201u IPv4 10324145 TCP localhost:56256-
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday 202u IPv4 10324147 TCP localhost:56257-
> > >localhost:9306 (ESTABLISHED)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> > COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> > drizzled 20886 eday 10u IPv4 10302592 TCP *:9306 (LISTEN)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> > tcp 0 0 127.0.0.1:56243 127.0.0.1:9306
> > ESTABLISHED
> > tcp 0 0 127.0.0.1:56255 127.0.0.1:9306 ...

Also, FWIW, I can't reproduce this on Solaris, seems to just be Linux.

-Eric

On Fri, Nov 06, 2009 at 02:56:22AM -0000, Brian Aker wrote:
> Hi!
> 
> Odd are... bug in drizzleslap. I've not looked at it in sometime. It  
> is still using the my_getopt junk so it links to the mysys mess... so  
> it probably has stuff dinking with its threads.
> 
> Cheers,
> 	-Brian
> 
> On Nov 5, 2009, at 6:33 PM, Eric Day wrote:
> 
> > Something seems to be broken once we exceed the back_log count. Once
> > getting drizzleslap to hang, I dumped the active sockets for each
> > process:
> >
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> > COMMAND     PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> > drizzlesl 32531 eday    3u  IPv4 10320113       TCP localhost:54858- 
> > >localhost:9306 (CLOSE_WAIT)
> > drizzlesl 32531 eday  185u  IPv4 10324113       TCP localhost:56240- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  186u  IPv4 10324115       TCP localhost:56241- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  187u  IPv4 10324117       TCP localhost:56242- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  188u  IPv4 10324119       TCP localhost:56243- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  189u  IPv4 10324121       TCP localhost:56244- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  190u  IPv4 10324123       TCP localhost:56245- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  191u  IPv4 10324125       TCP localhost:56246- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  192u  IPv4 10324127       TCP localhost:56247- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  193u  IPv4 10324129       TCP localhost:56248- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  194u  IPv4 10324131       TCP localhost:56249- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  195u  IPv4 10324133       TCP localhost:56250- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  196u  IPv4 10324135       TCP localhost:56251- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  197u  IPv4 10324137       TCP localhost:56252- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  198u  IPv4 10324139       TCP localhost:56253- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  199u  IPv4 10324141       TCP localhost:56254- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  200u  IPv4 10324143       TCP localhost:56255- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  201u  IPv4 10324145       TCP localhost:56256- 
> > >localhost:9306 (ESTABLISHED)
> > drizzlesl 32531 eday  202u  IPv4 10324147       TCP localhost:56257- 
> > >localhost:9306 (ESTABLISHED)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> > COMMAND    PID USER   FD   TYPE   DEVICE SIZE NODE NAME
> > drizzled 20886 eday   10u  IPv4 10302592       TCP *:9306 (LISTEN)
> > eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> > tcp        0      0 127.0.0.1:56243         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56255         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        1      0 127.0.0.1:54858         127.0.0.1:9306           
> > CLOSE_WAIT
> > tcp        0      0 127.0.0.1:56247         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56252         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56249         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56256         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56251         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56240         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56257         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56254         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56245         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56241         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56246         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56242         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56248         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56250         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56244         127.0.0.1:9306           
> > ESTABLISHED
> > tcp        0      0 127.0.0.1:56253         127.0.0.1:9306           
> > ESTABLISHED
> >
> >
> > As you can see, drizzleslap thinks it has valid connections and are  
> > all blocking in poll(), but the server has no knowledge of the  
> > active connections. The OS should be causing poll() to return  
> > POLLERR or POLLHUP inside of drizzleslap, rather than have them  
> > block indefinitely for POLLIN.
> >
> > -- 
> > Drizzleslap hangs with over 175 connections
> > https://bugs.launchpad.net/bugs/455137
> > You received this bug notification because you are a member of  
> > Drizzle-
> > developers, which is subscribed to Drizzle.
> >
> > Status in A Lightweight SQL Database for Cloud and Web: Confirmed
> >
> > Bug description:
> > Running drizzleslap with any number connections great than 175  
> > causes it to hang:
> >
> > Example:
> > drizzleslap   --port=9306 --label=write --concurrency=200 -- 
> > iterations=10 --engine=innodb --number-int
> > -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql- 
> > add-autoincrement --auto-genera
> > te-sql-load-type=write --number-of-queries=100
> >
> > attaching to the process through gdb gives:
> > (gdb) bt
> > #0  0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 ()  
> > from /lib/libpthread.so.0
> > #1  0x00000000004051c2 in run_scheduler (sptr=0xa462e0,  
> > stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> > #2  0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80,  
> > current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> > #3  0x0000000000409101 in main (argc=0, argv=0xa44178) at client/ 
> > drizzleslap.cc:377
> >
> > adding -v -v -v for lots of verbose output shows that it hangs while  
> > doing a bunch of inserts:
> > ........
> > INSERT INTO t1 VALUES (NULL, 
> > 1576223060,942232734 
> > ,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH 
> > ','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l 
> > ','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1322166549,823399162 
> > ,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg 
> > ','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L 
> > ','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 116314197,805287600 
> > ,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4 
> > ','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3 
> > ','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1407841593,32683953 
> > ,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880 
> > ','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH 
> > ','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM 
> > ');
> > INSERT INTO t1 VALUES (NULL, 
> > 1471079933,266531426 
> > ,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc 
> > ','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn 
> > ','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O 
> > ');
> >
> > For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ 
> >  to only use a max of 175 connections
> 
> -- 
> Drizzleslap hangs with over 175 connections
> https://bugs.launchpad.net/bugs/455137
> You received this bug notification because you are a bug assignee.
> 
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
> 
> Bug description:
> Running drizzleslap with any number connections great than 175 causes it to hang:
> 
> Example:
> drizzleslap   --port=9306 --label=write --concurrency=200 --iterations=10 --engine=innodb --number-int
> -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-genera
> te-sql-load-type=write --number-of-queries=100
> 
> attaching to the process through gdb gives:
> (gdb) bt
> #0  0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
> #1  0x00000000004051c2 in run_scheduler (sptr=0xa462e0, stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> #2  0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80, current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> #3  0x0000000000409101 in main (argc=0, argv=0xa44178) at client/drizzleslap.cc:377
> 
> adding -v -v -v for lots of verbose output shows that it hangs while doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL,1576223060,942232734,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW');
> INSERT INTO t1 VALUES (NULL,1322166549,823399162,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF');
> INSERT INTO t1 VALUES (NULL,116314197,805287600,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn');
> INSERT INTO t1 VALUES (NULL,1407841593,32683953,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM');
> INSERT INTO t1 VALUES (NULL,1471079933,266531426,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O');
> 
> For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/ to only use a max of 175 connections

Revision history for this message

Eric Day (eday) wrote on 2009-12-07:

#8

Marking as low since there is a workaround in place. Need to look into this more on Linux, may not be Drizzle specific.

Changed in drizzle:
importance:	Critical → Low

Lee Bieber (kalebral-deactivatedaccount) on 2010-01-28

Changed in drizzle:
milestone:	bell → cherry

Lee Bieber (kalebral-deactivatedaccount) on 2010-05-18

Changed in drizzle:
milestone:	2010-05-24 → none
status:	Confirmed → Won't Fix

Lee Bieber (kalebral-deactivatedaccount) on 2010-08-09

Changed in drizzle:
assignee:	Eric Day (eday) → nobody

Lee Bieber (kalebral-deactivatedaccount) on 2010-08-24

Changed in drizzle:
status:	Won't Fix → Confirmed

Revision history for this message

Lee Bieber (kalebral-deactivatedaccount) wrote on 2010-11-03:

#9

Confirmed that this still happens for the following test at 200 connections, all other tests appear to be passing when using 200 connections

scan-scale
% drizzleslap --port=9306 --label=scan-scale --concurrency=200 --iterations=10 --csv=/tmp/drizzleslap-1900.csv --number-int-cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=read --auto-generate-sql-execute-number=1000 --engine=innodb
Wed Nov 3 01:04:59 2010 INFO: scan-scale Failed with 34304
(null): Cannot drop database 'drizzleslap' ERROR : drizzle_state_read:lost connection to server (EOF)

Lee Bieber (kalebral-deactivatedaccount) on 2010-12-28

Changed in drizzle:
assignee:	nobody → Brian Aker (brianaker)

Drizzle

Drizzleslap hangs with over 175 connections

Bug Description

Related branches

Other bug subscribers

Remote bug watches