on restart, contrail-control hogs CPU for hours on scale setup
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.20 |
Fix Committed
|
Medium
|
Tapan Karwa | |||
Trunk |
Fix Committed
|
Medium
|
Tapan Karwa |
Bug Description
R2.20 Build 45 Ubuntu 14.04 Juno multi-node setup
In this scale setup, there are 64 tor-agents on 4 tor-agent-nodes
On contrail-control restart on the three nodes..it was seen that one of the control nodes which serves xmpp config for large number of tor-agents hogs CPU for hours.
if there are no agents/tor-agents subscribed, control-node stabilizes in < 10mins
Then, Starting the tor-agents later also stabilizes the setup quickly (<10 min)
Per Nischal, most of the time is being spent in graph walk in control node
env.roledefs = {
'all': [host2, host3, host4, host5, host6, host7, host8, host9],
'cfgm': [host2, host3, host4],
'openstack': [host2, host3, host4],
'webui': [host3],
'control': [host2, host3, host4],
'compute': [host5, host6, host7, host8, host9],
'collector': [host2, host3, host4],
'database': [host2, host3, host4],
'toragent': [host5, host6, host7, host9 ],
'tsn': [host5, host6, host7,host9 ],
'build': [host_build],
}
env.hostnames = {
'all': ['nodei34', 'nodei35', 'nodei36', 'nodei37', 'nodei38', 'nodei28', 'nodei27', 'nodei30']
}
A partial fix is to modify control node behavior to not accept xmpp
connections till it has received all nodes and links from irond. This
heuristic/behavior is tracked by bug 1446869. This should prevent
multiple repeated graph walks.
Further improvements will require optimizing the graph walker to
reduce the amount of CPU used.
Both of these are non-trivial changes and will be addressed in the
next release i.e. 2.21.