"MySQL server has gone away" again
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Invalid
|
High
|
Unassigned | ||
oslo.db |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
This is a regression of an old issue, which I thought was resolved by the "SELECT 1;" hack, but perhaps recently reintroduced with oslo.db?
[Mon Aug 25 14:30:54.403538 2014] [:error] [pid 25778:tid 139886259214080] 25778 ERROR keystone.
[Mon Aug 25 14:30:54.403562 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403570 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403575 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403581 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403589 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403594 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403600 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403606 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403612 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403618 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403624 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403630 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403637 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403644 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403650 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403656 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403662 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403667 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403673 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403680 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403731 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403740 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403746 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403752 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403757 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403763 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403769 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403775 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403782 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403787 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403793 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403799 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403804 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403810 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403815 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403820 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403826 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403833 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403839 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403845 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403851 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403857 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403862 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403868 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403888 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403894 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403901 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403906 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403911 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403918 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.403924 2014] [:error] [pid 25778:tid 139886259214080] 25778 TRACE keystone.
[Mon Aug 25 14:30:54.425356 2014] [authz_core:debug] [pid 25788:tid 139886145304320] mod_authz_
[Mon Aug 25 14:30:54.425387 2014] [authz_core:debug] [pid 25788:tid 139886145304320] mod_authz_
[Mon Aug 25 14:30:54.425432 2014] [authz_core:debug] [pid 25788:tid 139886145304320] mod_authz_
[Mon Aug 25 14:30:54.425446 2014] [authz_core:debug] [pid 25788:tid 139886145304320] mod_authz_
[Mon Aug 25 14:30:54.426830 2014] [:error] [pid 25779:tid 139886259214080] 25779 DEBUG keystone.
[Mon Aug 25 14:30:54.428226 2014] [:error] [pid 25779:tid 139886259214080] 25779 DEBUG keystone.
[Mon Aug 25 14:30:54.432009 2014] [:error] [pid 25779:tid 139886259214080] 25779 ERROR keystone.
[Mon Aug 25 14:30:54.432034 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432041 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432047 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432053 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432059 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432065 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432071 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432091 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432097 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432103 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432108 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432114 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432121 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432128 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432135 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432140 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432146 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432154 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432160 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432169 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432174 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432180 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432186 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432191 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432197 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432203 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432209 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432217 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432223 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432232 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432250 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432256 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432263 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432270 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432277 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
[Mon Aug 25 14:30:54.432286 2014] [:error] [pid 25779:tid 139886259214080] 25779 TRACE keystone.
This appeared as a transient issue in http://
Changed in keystone: | |
importance: | Undecided → High |
tags: | added: db |
tags: | added: sql |
affects: | oslo-incubator → oslo.db |
Changed in keystone: | |
status: | New → Invalid |
oslo.db issues 'SELECT 1' unconditionally right after a new transaction begins:
https:/ /github. com/openstack/ oslo.db/ blob/master/ oslo/db/ sqlalchemy/ session. py#L382 /github. com/openstack/ oslo.db/ blob/master/ oslo/db/ sqlalchemy/ session. py#L314- L331
https:/
which allows to detect disconnects early and reconnect to RDMBs servers transparently to the application code.
At the same time, if an RDBMs server goes away in the middle of transaction, it's left up to the consuming application to retry the whole transaction again. oslo.db provides a helper decorator for this: https:/ /github. com/openstack/ oslo.db/ blob/master/ oslo/db/ api.py# L62-L93