OpenStack Compute (nova)

List AVZs can take several seconds

  1. Series rocky
  2. Bug #1801897
Bug #1801897 reported by Belmiro Moreira
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Matt Riedemann
Queens
Confirmed
Medium
Unassigned
Rocky
In Progress
Medium
Matt Riedemann
Stein
In Progress
Medium
Matt Riedemann

Bug Description

Getting the list of AVZs can take several seconds (~30 secs. in our case)
This is noticeable in Horizon when creating a new instance because the user can't select an AVZ until this completes.

workflow:
- get all services from all cells (~10000 for us)
- fetch all aggregates which are tagged as an AVZ
- construct a dict of {'service['host']: avz.value}
- return a dict of {'avz_value': list of hosts}
- separate available and not available zones.

Reproducible in Queens, Rocky

Surya Seetharaman (tssurya)
Changed in nova:
assignee: nobody → Surya Seetharaman (tssurya)
tags: added: availability-zones performance
Revision history for this message
Matt Riedemann (mriedem) wrote :

Hmm, we do do a scatter/gather here when listing services:

https://github.com/openstack/nova/blob/b93b40c6c01a1161f40592d29353c3461669de19/nova/compute/api.py#L5106

Would be good to know where the majority of the time is spent in the overall flow.

Changed in nova:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Andrey Volkov (avolkov) wrote :

There is a room for improvement:
1) service list is fetched twice
   https://github.com/openstack/nova/blob/master/nova/availability_zones.py#L127-L128
   https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/availability_zone.py#L63-L64
2) it's better to use set in
   https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/availability_zone.py#L74-L75
3) most important, from tcpdump I see services are fetched by one from
   some place, see (http://paste.openstack.org/show/745011/)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/636947

Revision history for this message
Andrey Volkov (avolkov) wrote :

3) assumption is wrong, services are not fetched by one.

Matt Riedemann (mriedem)
Changed in nova:
assignee: Surya Seetharaman (tssurya) → Andrey Volkov (avolkov)
status: Confirmed → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Andrey Volkov (avolkov) → Matt Riedemann (mriedem)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Looks like this could go back to pike:

https://review.opendev.org/#/c/442163/

Although ^ isn't probably the reason it's slow, it's probably just always been slow.

Changed in nova:
assignee: Matt Riedemann (mriedem) → Andrey Volkov (avolkov)
Revision history for this message
Matt Riedemann (mriedem) wrote :

Note that the scatter/gather routine when fetching services from the HostAPI was already backported to pike:

https://review.opendev.org/#/q/I90b488102eb265d971cade29892279a22d3b5273

Matt Riedemann (mriedem)
Changed in nova:
importance: High → Medium
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Andrey Volkov (avolkov) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)
Download full text (4.2 KiB)

Reviewed: https://review.opendev.org/636947
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=74cefe4266a613d4c2afbb0c791e16eb7789aef4
Submitter: Zuul
Branch: master

commit 74cefe4266a613d4c2afbb0c791e16eb7789aef4
Author: Andrey Volkov <email address hidden>
Date: Thu Feb 14 15:39:45 2019 +0300

    AZ list performance optimization: avoid double service list DB fetch

    Assume number of services can be large (10000 as in the bug description),
    this patch removes second service_get_all call.

    zone_hosts changed from dict of lists to dict of sets.

    The HostAPI instance from the API controller is also passed to the
    get_availability_zones method so it does not have to recreate it
    per call (this is both for a slight performance gain but mostly also
    for test sanity).

    On devstack with 10000 services patch decreased response time twice.

    openstack availability zone list --long --timing

    ...

    Before:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.006816 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.456708 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.087485 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 95.667192 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.036528 |
    | Total | 96.254729 |
    +-------------------------------------------------------------------------------------------+-----------+

    After:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.020215 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.102987 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.111899 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 39.346657 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.026403 |
    | Total | 39.608161 |
    +-------------------------------------------------------------------------------...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656382

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)
Download full text (4.3 KiB)

Reviewed: https://review.opendev.org/656382
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c280d747fb23f1abaaf91eea7f6d11e716c6db42
Submitter: Zuul
Branch: stable/stein

commit c280d747fb23f1abaaf91eea7f6d11e716c6db42
Author: Andrey Volkov <email address hidden>
Date: Thu Feb 14 15:39:45 2019 +0300

    AZ list performance optimization: avoid double service list DB fetch

    Assume number of services can be large (10000 as in the bug description),
    this patch removes second service_get_all call.

    zone_hosts changed from dict of lists to dict of sets.

    The HostAPI instance from the API controller is also passed to the
    get_availability_zones method so it does not have to recreate it
    per call (this is both for a slight performance gain but mostly also
    for test sanity).

    On devstack with 10000 services patch decreased response time twice.

    openstack availability zone list --long --timing

    ...

    Before:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.006816 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.456708 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.087485 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 95.667192 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.036528 |
    | Total | 96.254729 |
    +-------------------------------------------------------------------------------------------+-----------+

    After:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.020215 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.102987 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.111899 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 39.346657 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.026403 |
    | Total | 39.608161 |
    +-------------------------------------------------------------------------...

Read more...

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/656510

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)
Download full text (4.4 KiB)

Reviewed: https://review.opendev.org/656510
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fa275544c27a5b428ad34b4c03be511607d15bf6
Submitter: Zuul
Branch: stable/rocky

commit fa275544c27a5b428ad34b4c03be511607d15bf6
Author: Andrey Volkov <email address hidden>
Date: Thu Feb 14 15:39:45 2019 +0300

    AZ list performance optimization: avoid double service list DB fetch

    Assume number of services can be large (10000 as in the bug description),
    this patch removes second service_get_all call.

    zone_hosts changed from dict of lists to dict of sets.

    The HostAPI instance from the API controller is also passed to the
    get_availability_zones method so it does not have to recreate it
    per call (this is both for a slight performance gain but mostly also
    for test sanity).

    On devstack with 10000 services patch decreased response time twice.

    openstack availability zone list --long --timing

    ...

    Before:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.006816 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.456708 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.087485 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 95.667192 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.036528 |
    | Total | 96.254729 |
    +-------------------------------------------------------------------------------------------+-----------+

    After:

    +-------------------------------------------------------------------------------------------+-----------+
    | URL | Seconds |
    +-------------------------------------------------------------------------------------------+-----------+
    | GET http://192.168.0.45/identity | 0.020215 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.102987 |
    | POST http://192.168.0.45/identity/v3/auth/tokens | 0.111899 |
    | GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 39.346657 |
    | GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.026403 |
    | Total | 39.608161 |
    +-------------------------------------------------------------------------...

Read more...

tags: added: in-stable-rocky
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.