Comment 5 for bug 455137

Revision history for this message
Brian Aker (brianaker) wrote : Re: [Bug 455137] Re: Drizzleslap hangs with over 175 connections

Hi!

Odd are... bug in drizzleslap. I've not looked at it in sometime. It
is still using the my_getopt junk so it links to the mysys mess... so
it probably has stuff dinking with its threads.

Cheers,
 -Brian

On Nov 5, 2009, at 6:33 PM, Eric Day wrote:

> Something seems to be broken once we exceed the back_log count. Once
> getting drizzleslap to hang, I dumped the active sockets for each
> process:
>
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 32531 -i4 -a
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzlesl 32531 eday 3u IPv4 10320113 TCP localhost:54858-
> >localhost:9306 (CLOSE_WAIT)
> drizzlesl 32531 eday 185u IPv4 10324113 TCP localhost:56240-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 186u IPv4 10324115 TCP localhost:56241-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 187u IPv4 10324117 TCP localhost:56242-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 188u IPv4 10324119 TCP localhost:56243-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 189u IPv4 10324121 TCP localhost:56244-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 190u IPv4 10324123 TCP localhost:56245-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 191u IPv4 10324125 TCP localhost:56246-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 192u IPv4 10324127 TCP localhost:56247-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 193u IPv4 10324129 TCP localhost:56248-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 194u IPv4 10324131 TCP localhost:56249-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 195u IPv4 10324133 TCP localhost:56250-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 196u IPv4 10324135 TCP localhost:56251-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 197u IPv4 10324137 TCP localhost:56252-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 198u IPv4 10324139 TCP localhost:56253-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 199u IPv4 10324141 TCP localhost:56254-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 200u IPv4 10324143 TCP localhost:56255-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 201u IPv4 10324145 TCP localhost:56256-
> >localhost:9306 (ESTABLISHED)
> drizzlesl 32531 eday 202u IPv4 10324147 TCP localhost:56257-
> >localhost:9306 (ESTABLISHED)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ lsof -p 20886 -i4 -a
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
> drizzled 20886 eday 10u IPv4 10302592 TCP *:9306 (LISTEN)
> eday@orisndriz01:~/bzr/drizzle/eday-dev/tests$ netstat -n|grep 9306
> tcp 0 0 127.0.0.1:56243 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56255 127.0.0.1:9306
> ESTABLISHED
> tcp 1 0 127.0.0.1:54858 127.0.0.1:9306
> CLOSE_WAIT
> tcp 0 0 127.0.0.1:56247 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56252 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56249 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56256 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56251 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56240 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56257 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56254 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56245 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56241 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56246 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56242 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56248 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56250 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56244 127.0.0.1:9306
> ESTABLISHED
> tcp 0 0 127.0.0.1:56253 127.0.0.1:9306
> ESTABLISHED
>
>
> As you can see, drizzleslap thinks it has valid connections and are
> all blocking in poll(), but the server has no knowledge of the
> active connections. The OS should be causing poll() to return
> POLLERR or POLLHUP inside of drizzleslap, rather than have them
> block indefinitely for POLLIN.
>
> --
> Drizzleslap hangs with over 175 connections
> https://bugs.launchpad.net/bugs/455137
> You received this bug notification because you are a member of
> Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
>
> Bug description:
> Running drizzleslap with any number connections great than 175
> causes it to hang:
>
> Example:
> drizzleslap --port=9306 --label=write --concurrency=200 --
> iterations=10 --engine=innodb --number-int
> -cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-
> add-autoincrement --auto-genera
> te-sql-load-type=write --number-of-queries=100
>
> attaching to the process through gdb gives:
> (gdb) bt
> #0 0x00007f3c0d98756d in pthread_cond_timedwait@@GLIBC_2.3.2 ()
> from /lib/libpthread.so.0
> #1 0x00000000004051c2 in run_scheduler (sptr=0xa462e0,
> stmts=0xa44350, concur=200, limit=0) at client/drizzleslap.cc:2137
> #2 0x0000000000408e12 in concurrency_loop (con=0x7fff16fdbe80,
> current=200, eptr=0xa45970) at client/drizzleslap.cc:495
> #3 0x0000000000409101 in main (argc=0, argv=0xa44178) at client/
> drizzleslap.cc:377
>
> adding -v -v -v for lots of verbose output shows that it hangs while
> doing a bunch of inserts:
> ........
> INSERT INTO t1 VALUES (NULL,
> 1576223060,942232734
> ,'nHujWOi8H3SwwB0cewKe8d5AyZL6EZ01qmkDAILKLh89t8lXwx9xQ67OXKWltWek9EpBYAXljfvcfY0cM1Zo8YDXsbHlynL7uAIQCi4wDrOtFOTdQuQQQXpI6y3wmH
> ','3gREyYw2S9ls219NeRHwlbRatQZpMJ6IrPYppKkHTT2OWBf1WpxFOHpAbMzpZx7OEX64rqDDBFdaIGbo5O3MhlYG7XyZK5xQTvWKlcZxj2X2JyiGMee4NACMZ2vl8l
> ','4dgQpQ2FFmHegccMlq0FmNIrmjthL1IPWzrCHuRYgk2KDWaqKb57qNyczrAKlSl7JcJcMe8uN26SYhItGF0yXybWhdqQxRQHvltIGuCfwJzTs9M0OEyv4PHl274sQW
> ');
> INSERT INTO t1 VALUES (NULL,
> 1322166549,823399162
> ,'7ugNokudvtyn2CnJQmEMBO76N3yqZzCZtkwaSrBGc93XDjqfLwtOK0WkwKaLJeu5MrfiaiyAjuZwdNRyKKMYClG85iMHKGMaztHPR7SA11yWPPKddj9pKQqH8CQLKg
> ','vKZ4d0C52570cOHw1uXAkjSQ1TeJqy0lAPqoQ2lKzsLR8WG9PnC1NYRG1xbrvbdy1ReJLNfC8sgGFwIgbMhyu0EwxinsHGiAbKLoct1klhs0d2Y4O53J5972run13L
> ','3eiPRAISuv1eoWhKaxQTGNfFIMA5F5QJj0k2A2WwxNRlt8yS5GpEfSTxjXuycD9CDtEGnd4d0XEufCEdS3j0O4p7ckx4x7G2sNAg0FH0mXmIc0nwwWoKaeJ2E873FF
> ');
> INSERT INTO t1 VALUES (NULL,
> 116314197,805287600
> ,'e8YWNpN1C9JgA6T6SHI3vdx1lw4tC2liAH5PySQ2dejfcCd5Lv0HOxA2uFnyHYYJhdcf52ZbYGE9KkEifErw31qx8dnH1KSax4DvzmWXvcY7KmhssOowIWlQzZc0l4
> ','Qi9fT8HrX4S4CoilYbACNSi0mhPMZARzlcWEcg6zkQ3o6eafDcrS4QSq7u5Z4YYh0vwSbuRBKN0QtQ562pQyFmgEYlnT9CA176Ta0plC4dfxvkvpZDngzKvY5it7m3
> ','8t2WJuJx7FamCMMzBoCrSBBFb8ySFkYGWQmhKxeRC55ORSNWGbLceYsEhiaoIy4gOiMlfs4rxA7GeMnmp0ESZ6a6g8mz7jXNPt85mDwbN4ktIXf8XlcwkAuOAZF9Gn
> ');
> INSERT INTO t1 VALUES (NULL,
> 1407841593,32683953
> ,'YZeAmRm1NWuhtPpG3P5nRrBWOJXYy46WWkhG3JAQhw8QEpaH7f4QN7wfISCGOInuuK3qX5G41GM76wG5SCXrKKMWgan4kQrXmmL9rTDkmbskN8prLCaZYx3CNiH880
> ','XumsvTL8Dzbxbz6GiJTHk16n6TMFdWF8HtsCM6DRxoPZfNhEiAXQ4T5AyRPSyYS7PLJfJO6G5XpkuXycZLuTqsfhBw91Sd8lqSO9sXQx27apWOHweCPKw517t29EgH
> ','zMdGYMnyJhxlWS2gOgsp0gmunYwwmSEdrrtFDg5PE22cM4Gek0Kdg6XSWJGGoWuXEneJKjcQlf20jBeKBypkxK4RXKkBrWax9CGTvlvZOqZz13BC20orCsIcDunwGM
> ');
> INSERT INTO t1 VALUES (NULL,
> 1471079933,266531426
> ,'a1vMmjn2ZKuaFxe9yR02BJeO5IKMg5o67bKlI7gjJQHbNokD7lGIwmjt4vFc1vb0Xvdh3HPEbipzMQCT3Koz7Ns3I0gJv7JQ2x8x6P4i7tZMBlrEyXDxuysDqOOlXc
> ','4QZ4NXTJ5SDeqOGaXE5k30av5OKT2r56HW2ZuXszrxeiEm8lsDLv5oiA42g6md4v86K2T5tLuNv893l18zoEfNOjPwq3amratBSE8Db3dM3eHhYQgKgJjwu1lkwJXn
> ','lGr5WzIy2vC6aWn6oJIKWuGOnt1jCOXpYMu2LDsG85EaRtYFCrbadjyOdr7hFWXfkHiyWQEXNKxqDW5HDYrq8h6dP5AYcZCKrmADCGkccaQp4O69opzoN5PoASCe1O
> ');
>
> For now modifying all tests at http://hudson.drizzle.org/view/Drizzle-staging/job/Drizzleslap-staging/
> to only use a max of 175 connections