large number of interfaces cause slow puppet executions due to fact generation

[Alex Schultz]

When a compute node (or controller) has many interfaces (bridges/tun), during the application of updates the puppet processes start thrashing trying to generate the network puppet facts. This affects both facter 2 and facter 3 based systems due to the fact that multiple processes end up trying to execute ip commands to gather the networking facts. This can be worked around by caching the facts for the life of the container-puppet.py (or docker-puppet.py) executions. This affects all versions of TripleO since Ocata. Prior to Ocata, facter 2 is super slow when there are more than than 1000 interfaces on a host. At ~1500, it can take > 10 minutes to generate the facts.

for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done

$ time facter

facter2
real 9m51.817s
user 7m8.936s
sys 2m42.702s

facter3
real 0m2.954s
user 0m1.111s
sys 0m1.721s

$ time puppet facts

facter2
real 12m10.936s
user 8m16.478s
sys 3m54.138s

facter3
real 0m11.169s
user 0m5.522s
sys 0m4.002s

[Alex Schultz (alex-schultz)]

https://review.opendev.org/#/c/669011/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669011
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4ec57c60f30f994d8bf36021ff12670e6c8c77b5
Submitter: Zuul
Branch: master

commit 4ec57c60f30f994d8bf36021ff12670e6c8c77b5
Author: Alex Schultz <email address hidden>
Date: Wed Jul 3 19:02:20 2019 -0600

    Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    Closes-Bug: #1835959
    Change-Id: I6ee3849ff380304a184638493f33882b813bfdff

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/671492

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/671499

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/671500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671524

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671524
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=08e34dbcaebb0608df61d6540d4919cab71632f5
Submitter: Zuul
Branch: master

commit 08e34dbcaebb0608df61d6540d4919cab71632f5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/671492
Reason: we reverted this version.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/671499
Reason: we reverted this version.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/671500
Reason: we reverted this version.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/672047

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/672048

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/672049

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672831

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672831
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=27dbca1433472d24be80711ae367707b29b29cc4
Submitter: Zuul
Branch: master

commit 27dbca1433472d24be80711ae367707b29b29cc4
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/673115

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/673116

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/673117

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672049
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Submitter: Zuul
Branch: stable/queens

commit ef01df9b8df79f7faf21791e9d9ddcd85e706bd4
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673117
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=df1d2024210cea3942d97deebd10e0f1506227b5
Submitter: Zuul
Branch: stable/queens

commit df1d2024210cea3942d97deebd10e0f1506227b5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672048
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c6a76f9263c52320e24f000ca0c46b643f2d2852
Submitter: Zuul
Branch: stable/rocky

commit c6a76f9263c52320e24f000ca0c46b643f2d2852
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673116
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f55b8c736217a30290d296cf9c1573bdf68aab0c
Submitter: Zuul
Branch: stable/rocky

commit f55b8c736217a30290d296cf9c1573bdf68aab0c
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672047
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=10e8506e165a8aa8be7512dcd095bc661f7a347b
Submitter: Zuul
Branch: stable/stein

commit 10e8506e165a8aa8be7512dcd095bc661f7a347b
Author: Alex Schultz <email address hidden>
Date: Thu Jul 18 10:36:05 2019 -0600

    Re-Add facter cache for container configurations

    Rather than running all the facts for every invocation of
    container-puppet.py, let's cache the facts for the entire container
    config step. This is similar to what used to happen when we ran puppet
    in a single instance for a given step. This improves the time it takes
    puppet tp execute within the containers. When systems have large number
    of interfaces, the fact generation for puppet can come to a crawl given
    that puppet is not supposed to be invoked multiple times in parallel. By
    pre-caching the facts, we eliminate the need for puppet to do the
    network (and other fact) related lookups when running container
    configuration generation.

    This patch includes logic to exclude the puppet caching dirs that we
    mount as read only in the config containers.

    Conflicts:
     common/container-puppet.py

    Change-Id: I94002572ec7ac1767fbfbfdf4c6fa6fbf0cd58d5
    Closes-Bug: #1837082
    Closes-Bug: #1835959
    (cherry picked from commit 08e34dbcaebb0608df61d6540d4919cab71632f5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673115
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bf51f816b113a3908fad4e34c320f822e9891a5c
Submitter: Zuul
Branch: stable/stein

commit bf51f816b113a3908fad4e34c320f822e9891a5c
Author: Alex Schultz <email address hidden>
Date: Thu Jul 25 15:21:04 2019 -0600

    Set selinux type for facter.conf

    Change-Id: I2dda4643b5b25d4958a467c618d297267c46cfc1
    Related-Bug: #1835959
    (cherry picked from commit 27dbca1433472d24be80711ae367707b29b29cc4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.1 release.