Notify resource_versions from agents only when needed
resource_versions were included into agent state reports recently to
support rolling upgrades (commit 97a272a892fcf488949eeec4959156618caccae8)
The downside is that it brought additional processing when handling state
reports on server side: update of local resources versions cache and
more seriously rpc casts to all other servers to do the same.
All this led to a visible performance degradation at scale with hundreds
of agents constantly sending reports. Under load (rally test) agents
may start "blinking" which makes cluster very unstable.
In fact there is no need to send and update resource_versions in each state
report. I see two cases when it should be done:
1) agent was restarted (after it was upgraded);
2) agent revived - which means that server was not receiving or being able
to process state reports for some time (agent_down_time). During that
time agent might be upgraded and restarted.
So this patch makes agents include resource_versions info only on startup.
After agent revival server itself will update version_manager with
resource_versions taken from agent DB record - this is to avoid
version_manager being outdated.
Reviewed: https:/ /review. openstack. org/302792 /git.openstack. org/cgit/ openstack/ neutron/ commit/ ?id=e532ee3fccd 0820f9ab0efc417 ee787fb8c870e9
Committed: https:/
Submitter: Jenkins
Branch: master
commit e532ee3fccd0820 f9ab0efc417ee78 7fb8c870e9
Author: Oleg Bondarev <email address hidden>
Date: Thu Apr 7 16:45:52 2016 +0300
Notify resource_versions from agents only when needed
resource_ versions were included into agent state reports recently to 8949eeec4959156 618caccae8)
support rolling upgrades (commit 97a272a892fcf48
The downside is that it brought additional processing when handling state
reports on server side: update of local resources versions cache and
more seriously rpc casts to all other servers to do the same.
All this led to a visible performance degradation at scale with hundreds
of agents constantly sending reports. Under load (rally test) agents
may start "blinking" which makes cluster very unstable.
In fact there is no need to send and update resource_versions in each state
report. I see two cases when it should be done:
1) agent was restarted (after it was upgraded);
2) agent revived - which means that server was not receiving or being able
to process state reports for some time (agent_down_time). During that
time agent might be upgraded and restarted.
So this patch makes agents include resource_versions info only on startup. versions taken from agent DB record - this is to avoid
After agent revival server itself will update version_manager with
resource_
version_manager being outdated.
Closes-Bug: #1567497 f8af2a656749166 b6fb49bcd3b
Change-Id: I47a9869801f4e8