Map instances to new Cell1 takes excessive amounts of time to run on upgraded cloud

Bug #1729661 reported by Logan V
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Wishlist
Unassigned

Bug Description

Map instances to new Cell1 runs unconditionally every time the nova role is run. On a prod environment upgrading from Newton, this is taking 6+ minutes each run. I think running the "cell_v2 map_instances" is probably a one time migration? So we could maybe set a local fact to skip it after it has migrated the instances into the cellsv2 successfully one time.

Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (master)

Reviewed: https://review.openstack.org/547072
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=6321cf90304023c20dc6abb87e2fa20107b7051d
Submitter: Zuul
Branch: master

commit 6321cf90304023c20dc6abb87e2fa20107b7051d
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

    Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

    After a new compute host is built, it will need to mapped to a cell
    through the 'discover_hosts' command of nova-manage. This command will
    still need to be run against a host that has direct database access, but
    the command is able to handle mapping of all compute hosts that do not
    currently belong to the cell so it only needs to be run once per play.

    There can be a short delay in the time between a compute service
    starting and becoming available to be discovered and mapped but, for
    most deployments, compute host discovery will be handled automatically
    as a periodic task.

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/552898

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/552899

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/552900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/pike)

Reviewed: https://review.openstack.org/552899
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=c581c2af7b2bfdf60fb4467d7d5b17bed32fc85f
Submitter: Zuul
Branch: stable/pike

commit c581c2af7b2bfdf60fb4467d7d5b17bed32fc85f
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

    Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

    After a new compute host is built, it will need to mapped to a cell
    through the 'discover_hosts' command of nova-manage. This command will
    still need to be run against a host that has direct database access, but
    the command is able to handle mapping of all compute hosts that do not
    currently belong to the cell so it only needs to be run once per play.

    There can be a short delay in the time between a compute service
    starting and becoming available to be discovered and mapped but, for
    most deployments, compute host discovery will be handled automatically
    as a periodic task.

    Conflicts:
    >------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/ocata)

Reviewed: https://review.openstack.org/552900
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=707398789403293aaedc1d78ddc4a164611366c7
Submitter: Zuul
Branch: stable/ocata

commit 707398789403293aaedc1d78ddc4a164611366c7
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

    Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

    After a new compute host is built, it will need to mapped to a cell
    through the 'discover_hosts' command of nova-manage. This command will
    still need to be run against a host that has direct database access, but
    the command is able to handle mapping of all compute hosts that do not
    currently belong to the cell so it only needs to be run once per play.

    There can be a short delay in the time between a compute service
    starting and becoming available to be discovered and mapped but, for
    most deployments, compute host discovery will be handled automatically
    as a periodic task.

    Conflicts:
    >------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/queens)

Reviewed: https://review.openstack.org/552898
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=d451edc2b83128e182e8885adf01f925ea2f98b6
Submitter: Zuul
Branch: stable/queens

commit d451edc2b83128e182e8885adf01f925ea2f98b6
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

    Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

    After a new compute host is built, it will need to mapped to a cell
    through the 'discover_hosts' command of nova-manage. This command will
    still need to be run against a host that has direct database access, but
    the command is able to handle mapping of all compute hosts that do not
    currently belong to the cell so it only needs to be run once per play.

    There can be a short delay in the time between a compute service
    starting and becoming available to be discovered and mapped but, for
    most deployments, compute host discovery will be handled automatically
    as a periodic task.

    Conflicts:
    >------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible (stable/pike)

Reviewed: https://review.openstack.org/554675
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=5ca262641f5a8182d0f87c413f919c47b43b27ac
Submitter: Zuul
Branch: stable/pike

commit 5ca262641f5a8182d0f87c413f919c47b43b27ac
Author: d34dh0r53 <email address hidden>
Date: Tue Mar 20 14:44:51 2018 -0500

    Update os_nova SHA

    This updates the os_nova SHA to pull in the Nova API downtime fixes for
    bug 1749037.

    Related-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540

    Depends-On: https://review.openstack.org/552899

    Change-Id: I3e21a4b741c88a72294175786bf66df8ab19e390

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/583583

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (master)

Reviewed: https://review.openstack.org/583583
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=0f24bb8bd45e6c3c4f1dc0fed1418c6be79525ee
Submitter: Zuul
Branch: master

commit 0f24bb8bd45e6c3c4f1dc0fed1418c6be79525ee
Author: Marc Gariepy <email address hidden>
Date: Wed Jul 18 09:56:47 2018 -0400

    Map instances to cell_v2 only once.

    If nova-manage cell_v2 create_cell returns 2, it's because the cell already
    exists and all instances are correctly mapped to the cell. The cell_v2
    map_instances doesn't need to run.

    Change-Id: I38002a18dff4dbd44bd3df9e1969c9d50ab12705
    Related-Bug: 1729661

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/585156

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/585157

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-os_nova (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/585158

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/queens)

Reviewed: https://review.openstack.org/585156
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=d0cc02fc82af1d6230ddcf8eb35f90671f40528a
Submitter: Zuul
Branch: stable/queens

commit d0cc02fc82af1d6230ddcf8eb35f90671f40528a
Author: Marc Gariepy <email address hidden>
Date: Wed Jul 18 09:56:47 2018 -0400

    Map instances to cell_v2 only once.

    If nova-manage cell_v2 create_cell returns 2, it's because the cell already
    exists and all instances are correctly mapped to the cell. The cell_v2
    map_instances doesn't need to run.

    Change-Id: I38002a18dff4dbd44bd3df9e1969c9d50ab12705
    Related-Bug: 1729661
    (cherry picked from commit 0f24bb8bd45e6c3c4f1dc0fed1418c6be79525ee)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/ocata)

Reviewed: https://review.openstack.org/585158
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=0726f5cdfb66770db16fd373cdb2f8c8421b0924
Submitter: Zuul
Branch: stable/ocata

commit 0726f5cdfb66770db16fd373cdb2f8c8421b0924
Author: Marc Gariepy <email address hidden>
Date: Wed Jul 18 09:56:47 2018 -0400

    Map instances to cell_v2 only once.

    If nova-manage cell_v2 create_cell returns 2, it's because the cell already
    exists and all instances are correctly mapped to the cell. The cell_v2
    map_instances doesn't need to run.

    Change-Id: I38002a18dff4dbd44bd3df9e1969c9d50ab12705
    Related-Bug: 1729661
    (cherry picked from commit 0f24bb8bd45e6c3c4f1dc0fed1418c6be79525ee)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_nova (stable/pike)

Reviewed: https://review.openstack.org/585157
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=7e0d6b5477d6218467dc2692c74e4df8ea07fca6
Submitter: Zuul
Branch: stable/pike

commit 7e0d6b5477d6218467dc2692c74e4df8ea07fca6
Author: Marc Gariepy <email address hidden>
Date: Wed Jul 18 09:56:47 2018 -0400

    Map instances to cell_v2 only once.

    If nova-manage cell_v2 create_cell returns 2, it's because the cell already
    exists and all instances are correctly mapped to the cell. The cell_v2
    map_instances doesn't need to run.

    Change-Id: I38002a18dff4dbd44bd3df9e1969c9d50ab12705
    Related-Bug: 1729661
    (cherry picked from commit 0f24bb8bd45e6c3c4f1dc0fed1418c6be79525ee)

Changed in openstack-ansible:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.