executemany with many values (>200) crashes

[Kane Kim]

executemany with many values crashes:

Traceback (most recent call last):
  File "/root/test.py", line 40, in insert_entries
    db.cursor().execute(sql)
  File "/root/lib/mysql/connector/cursor.py", line 314, in execute
    res = self.db().protocol.cmd_query(stmt)
  File "/root/lib/mysql/connector/protocol.py", line 136, in deco
    return func(*args, **kwargs)
  File "/root/lib/mysql/connector/protocol.py", line 473, in cmd_query
    self.conn.send(pkt) # Errors handled in _handle_error()
  File "/root/lib/mysql/connector/connection.py", line 78, in send
    raise errors.OperationalError('%s' % e)
OperationalError: [Errno 32] Broken pipe

[Kane Kim (kane-isturm)]

Forgot to add - happens only in combination with gevent (i think it's mostly used together).

[Geert JM Vanderkelen (geertjmvdk)]

Hello Kane,

Thanks for opening this bug report.
Could the table and some example data be uploaded?

How are you using gevent with Connector/Python?

Cheers,

Geert

[Kane Kim (kane-isturm)]

Hello Geert,

It's actually quite easy:
from gevent only:

monkey.patch_all(thread=False)

is used, and then:
db.cursor().executemany("""INSERT INTO user (email, name, list_id, fields) values (%s, %s, %s, %s)""", entries)

so, when there are many rows inserted (looks like it depends on data size, so if inserting long string it would crash earlier), it would crash with above error.

Thanks a lot.

[tormen (quickhelp)]

Hi Geert,

We ran into similar problems here and it was related to the mysql variable MAX_ALLOWED_PACKET
So we changed our wrapper functions for "insert" to use and respect this value: E.g. chop up the insert into multiple executes.

Knuth

[Geert JM Vanderkelen (geertjmvdk)]

Thanks Tormen, we'll have to make a better reporting of this problem.
Thing is that the original bug report says it only happens with gevent.

[Geert JM Vanderkelen (geertjmvdk)]

I think this was also a problem with sending and receiving big packets.
v0.3 should help with that.