Max retries exceeded with StaleDataError in standardattributes during live migration
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
New
|
Undecided
|
Ryan Tidwell |
Bug Description
I am observing StaleDataError on stable/pike during live migration, causing live migration to fail. It occurs when attempting to live-migrate a handful of VM's (5-6 VM's is all it takes) in rapid succession from the same source to the same target. This quick and dirty script is able to make the issue appear reliably:
for i in `openstack server list --all-projects --host <origin> -c ID -f value`; do openstack server migrate $i --live <target>; done
From the neutron server logs:
DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattrib
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api Traceback (most recent call last):
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api return f(*args, **kwargs)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api LOG.debug("Retry wrapper got retriable exception: %s", e)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api self.force_
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api six.reraise(
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api return f(*dup_args, **dup_kwargs)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api mech_context, attrs)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api db.clear_
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api return fn(*args, **kwargs)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/usr/lib64/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api self.gen.next()
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api yield resource
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/usr/lib64/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api self.gen.next()
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api self.session.
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api self._flush(
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api transaction.
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api compat.
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api flush_context.
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api rec.execute(self)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api uow
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api mapper, table, update)
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api File "/opt/stack/
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api (table.description, len(records), rows))
2019-01-24 09:57:34.959 255478 ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattrib
It appears that somewhere in clear_binding_
Changed in neutron: | |
assignee: | nobody → Ryan Tidwell (ryan-tidwell) |
You mentioned that it is trying to migrate from the same source to the same target. So we are just removing and adding it to the same node?