Floating ip assigning failed with Error: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None (HTTP 400)
Steps:
1. Create and deploy next cluster - Ubuntu, HA, Neutron with tunnelling, 3 controllers, 2 compute nodes
2. After deployment open health check tab and start tests
Actual result - Check network connectivity from instance via floating IP test failed with
BadRequest: Unable to associate floating ip 10.109.11.130 to fixed ip 10.0.7.4 for instance 75d081e4-9a62-4992-acfb-211041846a69. Error: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') None None (HTTP 400) (Request-ID: req-99c2bd02-7f99-436d-bc9d-ebb94ede742b) fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 177, in verify result = func(*args, **kwargs) File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 698, in _assign_floating_ip_to_instance self.fail('Can not assign floating ip to instance') File "/usr/lib/python2.6/site-packages/unittest2/case.py", line 415, in fail raise self.failureException(msg) AssertionError: Can not assign floating ip to instance
Test passed after second run.
{
"build_id": "257",
"build_number": "257",
"release_versions":
{
"2015.1.0-7.0":
{
"VERSION":
{
"build_id": "257",
"build_number": "257",
"api": "1.0",
"fuel-library_sha": "bc04a7092d92400c79e6ea6ede25e7b67c6a6355",
"nailgun_sha": "3189ccfb8c1dac888e351f535b03bdbc9d392406",
"feature_groups":
[
"mirantis"
],
"fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
"openstack_version": "2015.1.0-7.0",
"fuel-agent_sha": "1e8f38bbb864ed99aa8fe862b6367e82afec3263",
"production": "docker",
"python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
"astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c",
"fuel-ostf_sha": "644db51186dc23c9b27e9b5486c120c8363dc87c",
"release": "7.0",
"fuelmain_sha": "0e54d68392b359bc122e5bbba9249c729eeaf579"
}
}
},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "bc04a7092d92400c79e6ea6ede25e7b67c6a6355",
"nailgun_sha": "3189ccfb8c1dac888e351f535b03bdbc9d392406",
"feature_groups":
[
"mirantis"
],
"fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
"openstack_version": "2015.1.0-7.0",
"fuel-agent_sha": "1e8f38bbb864ed99aa8fe862b6367e82afec3263",
"production": "docker",
"python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
"astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c",
"fuel-ostf_sha": "644db51186dc23c9b27e9b5486c120c8363dc87c",
"release": "7.0",
"fuelmain_sha": "0e54d68392b359bc122e5bbba9249c729eeaf579"
}
Logs are attached
(OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') is a very typical error, which means that Galera is running in multi-master mode (in Galera terms, all nodes are equal, multi-master here simply means you've got connections to different MySQL servers) and concurrent processes are trying to update a single row on two different nodes simultaneously. Galera does not support write-intent InnoDB locks by design, so one or both of the transactions will fail with a bogus deadlock error, which effectively means that application has to resolve the conflict of a concurrent update and retry the transaction. See [1] for details.
Nova (and other OpenStack projects) already provides @_retry_on_deadlock to be applied to DB API methods, so that ones updating rows retry transactions on the deadlock error. The problem with it, is that you have to apply it to every single writer method to ensure all the deadlocks are handled properly - this is both tedious and error-prone. EngineFacade [2] of oslo.db seems to be the right place to implement transparent retries on deadlocks, but we have to update all the OpenStack projects to use it first.
At the same time, in MOS we *intentionally* deploy Galera in active-backup mode (configured in HAProxy), so that all connections go the same MySQL server at each moment of time. The only possible reason you still see such deadlocks in this case is that HAProxy thought that the active MySQL server became down for some time and promoted a backup server to be active (and switched back again, once the original server became online). Looks like both MySQL servers were up, it's just that the active one didn't respond in time on the health check. Due to connection pooling in OpenStack services we ran into situation, when you've got connections to multiple Galera nodes, i.e. effectively enabled multi-master setup.
One way to avoid that would be to close all backup connections, when the active server is back online [3], but that would also cause connection errors (and effectively abort all ongoing transactions handled by backup MySQL nodes, as HAProxy does not care about application level data here). So it's not really a good solution either.
[1] http://
[2] http://
[3] http://