create_admin ansible execution timesout when there are many network interfaces on a system

Bug #1869018 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

Similar to the issues we saw with puppet's facter, ansible also suffers when trying to run fact gathering on systems with many network interfaces. We see the create_admin workflow timeout due to the playbook taking too long because the fact gathering runs long. The create_admin tasks don't require any facts as the playbook is static so we should disable fact gathering

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.opendev.org/714966

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/714966
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=32bc45c293c3ad9e24a803dfd82042efa81efdfe
Submitter: Zuul
Branch: master

commit 32bc45c293c3ad9e24a803dfd82042efa81efdfe
Author: Alex Schultz <email address hidden>
Date: Wed Mar 25 09:04:09 2020 -0600

    Disable fact gathering for create_admin

    The create_admin execution doesn't require facts as the tasks are
    static. Fact gather can be a long time on systems with large number of
    network interfaces or storage volumes. This can lead to timeouts when
    doing actions like scaling or deploying.

    Change-Id: I0ff270dfabab2ef6926b8d2818ef5087eb03bed9
    Closes-Bug: #1869018

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/715205

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/715206

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/715207

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/715208

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/queens)

Reviewed: https://review.opendev.org/715208
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=244ff4e071a82d3cede3c3e9c3bfedcc162e126e
Submitter: Zuul
Branch: stable/queens

commit 244ff4e071a82d3cede3c3e9c3bfedcc162e126e
Author: Alex Schultz <email address hidden>
Date: Wed Mar 25 09:04:09 2020 -0600

    Disable fact gathering for create_admin

    The create_admin execution doesn't require facts as the tasks are
    static. Fact gather can be a long time on systems with large number of
    network interfaces or storage volumes. This can lead to timeouts when
    doing actions like scaling or deploying.

    Change-Id: I0ff270dfabab2ef6926b8d2818ef5087eb03bed9
    Closes-Bug: #1869018
    (cherry picked from commit 32bc45c293c3ad9e24a803dfd82042efa81efdfe)

tags: added: in-stable-queens
tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/rocky)

Reviewed: https://review.opendev.org/715207
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5cd39ae6a0f81814baf3fbca873a3d22a21328b0
Submitter: Zuul
Branch: stable/rocky

commit 5cd39ae6a0f81814baf3fbca873a3d22a21328b0
Author: Alex Schultz <email address hidden>
Date: Wed Mar 25 09:04:09 2020 -0600

    Disable fact gathering for create_admin

    The create_admin execution doesn't require facts as the tasks are
    static. Fact gather can be a long time on systems with large number of
    network interfaces or storage volumes. This can lead to timeouts when
    doing actions like scaling or deploying.

    Change-Id: I0ff270dfabab2ef6926b8d2818ef5087eb03bed9
    Closes-Bug: #1869018
    (cherry picked from commit 32bc45c293c3ad9e24a803dfd82042efa81efdfe)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/train)

Reviewed: https://review.opendev.org/715205
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=eec15ed2dc65a8e7ee4df0ec13863a3e237111eb
Submitter: Zuul
Branch: stable/train

commit eec15ed2dc65a8e7ee4df0ec13863a3e237111eb
Author: Alex Schultz <email address hidden>
Date: Wed Mar 25 09:04:09 2020 -0600

    Disable fact gathering for create_admin

    The create_admin execution doesn't require facts as the tasks are
    static. Fact gather can be a long time on systems with large number of
    network interfaces or storage volumes. This can lead to timeouts when
    doing actions like scaling or deploying.

    Change-Id: I0ff270dfabab2ef6926b8d2818ef5087eb03bed9
    Closes-Bug: #1869018
    (cherry picked from commit 32bc45c293c3ad9e24a803dfd82042efa81efdfe)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/stein)

Reviewed: https://review.opendev.org/715206
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=41ad911593445c5856e870c1f6cf61c12e5cbc82
Submitter: Zuul
Branch: stable/stein

commit 41ad911593445c5856e870c1f6cf61c12e5cbc82
Author: Alex Schultz <email address hidden>
Date: Wed Mar 25 09:04:09 2020 -0600

    Disable fact gathering for create_admin

    The create_admin execution doesn't require facts as the tasks are
    static. Fact gather can be a long time on systems with large number of
    network interfaces or storage volumes. This can lead to timeouts when
    doing actions like scaling or deploying.

    Change-Id: I0ff270dfabab2ef6926b8d2818ef5087eb03bed9
    Closes-Bug: #1869018
    (cherry picked from commit 32bc45c293c3ad9e24a803dfd82042efa81efdfe)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 12.3.0

This issue was fixed in the openstack/tripleo-common 12.3.0 release.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

It may be worth improving facts caching and/or tripleo-ansible's and t-h-t's gather_facts cases, like

http://codesearch.openstack.org/?q=tripleo_target_host&i=nope&files=&repos=
and all that we have in tht common/deploy-steps.j2 and common/deploy-steps-playbooks-common.yaml

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common rocky-eol

This issue was fixed in the openstack/tripleo-common rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common queens-eol

This issue was fixed in the openstack/tripleo-common queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common stein-eol

This issue was fixed in the openstack/tripleo-common stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.