[cellv2] the performance issue of cellv2 when creating 500 instances concurrently
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | OpenStack Compute (nova) |
Medium
|
Unassigned | ||
Bug Description
Description
===========
we test cellv2 solution and execute creating instances concurrently.
But there is the performance issue due to query instance info in scheduler process
Steps to reproduce
==================
use rally to execute creating 500 instances concurrently
Expected result
===============
all instances are created successfully
Actual result
=============
many instances failed due to timeout
Environment
===========
1. Exact version of OpenStack you are running. See the following
openstack-
openstack-
2. Which hypervisor did you use?
KVM. We have 2000 hosts
2. Which storage type did you use?
Our own storage device
3. Which networking type did you use?
Our own network device
Analysis
==============
In scheduler process, get_host_
_get_host_state call host_state.update and _get_instance_info
So _get_instance_info will be called at every request and every hosts.
Because cellv2 have disable instance info cache.
So there's a performance issue even we don't use affinity filter policy
| Changed in nova: | |
| assignee: | nobody → Surya Seetharaman (tssurya) |
| tags: | added: cells scheduler |
| melanie witt (melwitt) wrote : | #1 |
| melanie witt (melwitt) wrote : | #2 |
To be clear, if the cache was disabled, it was unintentional and it is a bug.
| Jiang (jiangpf) wrote : | #3 |
This is a performance issue. Compute instances info just used by affinity filter.
Anti affinity will judgment the instance whether it belongs to the current computing node. The current code is to determine if the instance uuid is in the instances of the compute node.
We can determine if the current instance exists. If it exists, we get the host it belongs to. Then we judge in anti- affinity whether the host name of the compute node is the same as the instance host name.
| Changed in nova: | |
| assignee: | Surya Seetharaman (tssurya) → Jiang (jiangpf) |
| status: | New → Confirmed |
| Matt Riedemann (mriedem) wrote : | #4 |
By "Because cellv2 have disable instance info cache." I assume you mean you have disabled the track_instance_
| Changed in nova: | |
| assignee: | Jiang (jiangpf) → Matt Riedemann (mriedem) |
| importance: | Undecided → Medium |
| Matt Riedemann (mriedem) wrote : | #5 |
By the way, "track_
https:/
Fix proposed to branch: master
Review: https:/
| Changed in nova: | |
| status: | Confirmed → In Progress |
| Matt Riedemann (mriedem) wrote : | #7 |
You said, "So there's a performance issue even we don't use affinity filter policy" - are you actually not using any of the ServerGroup(
That would break any out of tree filters that rely on the HostState.instances information, however, so we'd have to consider that if it were a change we made.
Related fix proposed to branch: master
Review: https:/
| Changed in nova: | |
| assignee: | Matt Riedemann (mriedem) → Eric Fried (efried) |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit a1a335f19a986a5
Author: Matt Riedemann <email address hidden>
Date: Thu May 17 14:36:23 2018 -0400
Avoid unnecessary joins in HostManager.
While getting a HostState object for a given compute node during
scheduling, if the HostState does not have its instance info
set, either because it's out of date or because config option
"track_
the list of instances for that host from the database and stores
it in HostState.
This is *only* used (in-tree) by the affinity filters and even
then the only thing those filters use from HostState.instances
is the set of keys from the dict, which is the list of instance
UUIDs on a given host. The actual Instance objects aren't used
at all. See blueprint put-host-
for more details on that.
The point of this change, is that when we go to pull the set
of instances from the database for a given host, we don't need
to join on the default columns (info_cache and security_groups)
defined in the _instance_
This should be at least some minor optimization in scheduling
for hosts that have several instances on them in a large cloud.
As noted in the comment in the code, any out of tree filters
that rely on using the info_cache or security_groups from the
instance are now going to be hit with a lazy-load penalty
per instance, but we have no contract on out of tree filters
so if this happens, the people maintaining said filters can
(1) live with it (2) fork the HostManager code or (3) upstream
their filter so it's in-tree.
A more impactful change would be to refactor
HostManager
the given set of compute nodes in a single query per cell. But
that is left for a later change.
Change-Id: Iccefbfdfa57851
Partial-Bug: #1737465
| Changed in nova: | |
| assignee: | Eric Fried (efried) → Matt Riedemann (mriedem) |
Fix proposed to branch: stable/queens
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/queens
commit bfb9d3f179607c2
Author: Matt Riedemann <email address hidden>
Date: Thu May 17 14:36:23 2018 -0400
Avoid unnecessary joins in HostManager.
While getting a HostState object for a given compute node during
scheduling, if the HostState does not have its instance info
set, either because it's out of date or because config option
"track_
the list of instances for that host from the database and stores
it in HostState.
This is *only* used (in-tree) by the affinity filters and even
then the only thing those filters use from HostState.instances
is the set of keys from the dict, which is the list of instance
UUIDs on a given host. The actual Instance objects aren't used
at all. See blueprint put-host-
for more details on that.
The point of this change, is that when we go to pull the set
of instances from the database for a given host, we don't need
to join on the default columns (info_cache and security_groups)
defined in the _instance_
This should be at least some minor optimization in scheduling
for hosts that have several instances on them in a large cloud.
As noted in the comment in the code, any out of tree filters
that rely on using the info_cache or security_groups from the
instance are now going to be hit with a lazy-load penalty
per instance, but we have no contract on out of tree filters
so if this happens, the people maintaining said filters can
(1) live with it (2) fork the HostManager code or (3) upstream
their filter so it's in-tree.
A more impactful change would be to refactor
HostManager
the given set of compute nodes in a single query per cell. But
that is left for a later change.
Change-Id: Iccefbfdfa57851
Partial-Bug: #1737465
(cherry picked from commit a1a335f19a986a5
| tags: | added: in-stable-queens |
Related fix proposed to branch: master
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 91f5af7ee7f7140
Author: Matt Riedemann <email address hidden>
Date: Thu May 17 16:06:39 2018 -0400
Trim the fat on HostState.instances
The only in-tree filters that rely on HostState.instances
are the affinity filters (and one of the weighers). And they
don't even look at the values in the HostState.instances dict,
just the keys (which are the instance uuids for the instances
on the host).
So rather than pull the full instance objects, we can just get
the list of instance uuids off the host and fake out the object.
Custom filters/weighers will still be able to lazy-load fields
on the Instance objects in HostState.instances if needed, but
it will mean a performance penalty due to the round trip to the
database per instance, per host. Out of tree filters/weighers
are encouraged to be contributed upstream.
Related to blueprint put-host-
Related-Bug: #1737465
Change-Id: I766bb5645e3b59
Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https:/
| Matt Riedemann (mriedem) wrote : | #15 |
I've pushed some related changes for this bug am removing myself as assignee since I think we still have some known performance issues during scheduling, specifically around this method to get all host states from all cells, and for each compute (host state) getting the aggregates and instances from each cell db per host:
| Changed in nova: | |
| assignee: | Matt Riedemann (mriedem) → nobody |
| status: | In Progress → Confirmed |
| Matt Riedemann (mriedem) wrote : | #16 |
Some related changes:
1. Jay Pipes' patch https:/
2. Surya's patch https:/
Related fix proposed to branch: master
Review: https:/
Related fix proposed to branch: master
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 426a303ee4cb848
Author: Matt Riedemann <email address hidden>
Date: Wed Jun 5 12:54:45 2019 -0400
Convert HostMapping.cells to a dict
This changes the HostMapping.cells list to a dict,
keyed by cell uuid, to each CellMapping object.
This is going to be used in a later change [1] where
we will create a cached mapping of host to cell uuid
so we can go quickly from compute node host value to
get the CellMapping without having to lookup the
HostMapping again.
[1] Ic6b1edfad2e384
Change-Id: I75e5bf2c45688f
Related-Bug: #1737465
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 3a69fdb333e80f5
Author: Matt Riedemann <email address hidden>
Date: Wed Jun 5 13:30:25 2019 -0400
Cache host to cell mapping in HostManager
If the instances per host are not cached in the HostManager
we lookup the HostMapping per candidate compute node during
each scheduling request to get the CellMapping so we can target
that cell database to pull the instance uuids on the given host.
For example, if placement returned 20 compute node allocation
candidates and we don't have the instances cached for any of those,
we'll do 20 queries to the API DB to get host mappings.
We can improve this by caching the host to cell uuid after the first
lookup for a given host and then after that, get the CellMapping
from the cells cache (which is a dict, keyed by cell uuid, to the
CellMapping for that cell).
Change-Id: Ic6b1edfad2e384
Related-Bug: #1737465
Related fix proposed to branch: stable/stein
Review: https:/
Related fix proposed to branch: stable/stein
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 30552298410605e
Author: Matt Riedemann <email address hidden>
Date: Wed Jun 5 12:54:45 2019 -0400
Convert HostMapping.cells to a dict
This changes the HostMapping.cells list to a dict,
keyed by cell uuid, to each CellMapping object.
This is going to be used in a later change [1] where
we will create a cached mapping of host to cell uuid
so we can go quickly from compute node host value to
get the CellMapping without having to lookup the
HostMapping again.
[1] Ic6b1edfad2e384
Change-Id: I75e5bf2c45688f
Related-Bug: #1737465
(cherry picked from commit 426a303ee4cb848
| tags: | added: in-stable-stein |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 677c95280b21eb6
Author: Matt Riedemann <email address hidden>
Date: Wed Jun 5 13:30:25 2019 -0400
Cache host to cell mapping in HostManager
If the instances per host are not cached in the HostManager
we lookup the HostMapping per candidate compute node during
each scheduling request to get the CellMapping so we can target
that cell database to pull the instance uuids on the given host.
For example, if placement returned 20 compute node allocation
candidates and we don't have the instances cached for any of those,
we'll do 20 queries to the API DB to get host mappings.
We can improve this by caching the host to cell uuid after the first
lookup for a given host and then after that, get the CellMapping
from the cells cache (which is a dict, keyed by cell uuid, to the
CellMapping for that cell).
Change-Id: Ic6b1edfad2e384
Related-Bug: #1737465
(cherry picked from commit 3a69fdb333e80f5
Sorry for getting to this so long after it was opened. What do you mean by, "Because cellv2 have disable instance info cache."? I don't notice anything in the code that is deliberately disabling the instance info cache in the HostManager.