large number of interfaces cause slow puppet executions due to fact generation

Bug #1835959 reported by Alex Schultz on 2019-07-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Alex Schultz

Bug Description

When a compute node (or controller) has many interfaces (bridges/tun), during the application of updates the puppet processes start thrashing trying to generate the network puppet facts. This affects both facter 2 and facter 3 based systems due to the fact that multiple processes end up trying to execute ip commands to gather the networking facts. This can be worked around by caching the facts for the life of the container-puppet.py (or docker-puppet.py) executions. This affects all versions of TripleO since Ocata. Prior to Ocata, facter 2 is super slow when there are more than than 1000 interfaces on a host. At ~1500, it can take > 10 minutes to generate the facts.

for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done

$ time facter

facter2
real 9m51.817s
user 7m8.936s
sys 2m42.702s

facter3
real 0m2.954s
user 0m1.111s
sys 0m1.721s

$ time puppet facts

facter2
real 12m10.936s
user 8m16.478s
sys 3m54.138s

facter3
real 0m11.169s
user 0m5.522s
sys 0m4.002s

tags: added: queens-backport-potential rocky-backport-potential
Changed in tripleo:
milestone: none → train-2
Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.opendev.org/669011
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4ec57c60f30f994d8bf36021ff12670e6c8c77b5
Submitter: Zuul
Branch: master

commit 4ec57c60f30f994d8bf36021ff12670e6c8c77b5
Author: Alex Schultz <email address hidden>
Date: Wed Jul 3 19:02:20 2019 -0600

    Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    Closes-Bug: #1835959
    Change-Id: I6ee3849ff380304a184638493f33882b813bfdff

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/671524
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=08e34dbcaebb0608df61d6540d4919cab71632f5
Submitter: Zuul
Branch: master

commit 08e34dbcaebb0608df61d6540d4919cab71632f5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/671492
Reason: we reverted this version.

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/671499
Reason: we reverted this version.

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/671500
Reason: we reverted this version.

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

Reviewed: https://review.opendev.org/672831
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=27dbca1433472d24be80711ae367707b29b29cc4
Submitter: Zuul
Branch: master

commit 27dbca1433472d24be80711ae367707b29b29cc4
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959

Reviewed: https://review.opendev.org/672049
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Submitter: Zuul
Branch: stable/queens

commit ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

tags: added: in-stable-queens

Reviewed: https://review.opendev.org/673117
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=df1d2024210cea3942d97deebd10e0f1506227b5
Submitter: Zuul
Branch: stable/queens

commit df1d2024210cea3942d97deebd10e0f1506227b5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/672048
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c6a76f9263c52320e24f000ca0c46b643f2d2852
Submitter: Zuul
Branch: stable/rocky

commit c6a76f9263c52320e24f000ca0c46b643f2d2852
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

Reviewed: https://review.opendev.org/673116
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f55b8c736217a30290d296cf9c1573bdf68aab0c
Submitter: Zuul
Branch: stable/rocky

commit f55b8c736217a30290d296cf9c1573bdf68aab0c
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/672047
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=10e8506e165a8aa8be7512dcd095bc661f7a347b
Submitter: Zuul
Branch: stable/stein

commit 10e8506e165a8aa8be7512dcd095bc661f7a347b
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

Reviewed: https://review.opendev.org/673115
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bf51f816b113a3908fad4e34c320f822e9891a5c
Submitter: Zuul
Branch: stable/stein

commit bf51f816b113a3908fad4e34c320f822e9891a5c
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers