Ansible slowness due to a large number of network interface facts

[Rafael Weingartner]

This small patch speeds up kolla-ansible runs in production
environments with many network interfaces on the network and
compute nodes.

Before applying this patch, we tried the following tunnings:
* Increased the number of forks (great speedup depending of the size of
  the deployment)
* Use `strategy = mitogen_linear` (cut processing time in half)
* Ansible caching (little speed up)
* SSH tunning (little speed up)

The biggest speedup (about 18x) was achieved with the ansible facts
filtering. We guess that the best option, for now, is this patch. In
the future, we could get in touch with the ansible community to enable
extend facts filtering like, eg. allow users to define multiple filters
for each subset.

We experimented with this patch, using the following command.

```
EXTRA_OPTS='--check --diff' kolla-ansible -i inventory genconfig -t ceilometer
```

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/783392
Committed: https://opendev.org/openstack/kolla-ansible/commit/15f2fdcd5dd9e63b477c549c2af990df8235ca90
Submitter: "Zuul (22348)"
Branch: master

commit 15f2fdcd5dd9e63b477c549c2af990df8235ca90
Author: Rafael Weingärtner <email address hidden>
Date: Fri Mar 26 14:19:39 2021 -0300

    Make setup module arguments configurable

    Ansible facts can have a large impact on the performance of the Ansible
    control host. This patch introduces some control over which facts are
    gathered (kolla_ansible_setup_gather_subset) and which facts are stored
    (kolla_ansible_setup_filter). By default we do not change the default
    values of these arguments to the setup module. The flexibility of these
    arguments is limited, but they do provide enough for a large performance
    improvement in a typical moderate to large OpenStack cloud.

    In particular, the large complex dict fact for each interface has a
    large effect, and on an OpenStack controller or hypervisor there may be
    many virtual interfaces. We can use the kolla_ansible_setup_filter
    variable to help:

        kolla_ansible_setup_filter: 'ansible_[!qt]*'

    This causes Ansible to collect but not store facts matching that
    pattern, which includes the virtual interface facts. Currently we are
    not referencing other facts matching the pattern within Kolla Ansible.
    Note that including the 'ansible_' prefix causes meta facts module_setup
    and gather_subset to be filtered, but this seems to be the only way to
    get a good match on the interface facts. To work around this, we use
    ansible_facts rather than module_setup to detect whether facts exist in
    the cache.

    The exact improvement will vary, but has been reported to be as large as
    18x on systems with many virtual interfaces.

    For reference, here are some other tunings tried:

    * Increased the number of forks (great speedup depending of the size of
      the deployment)
    * Use `strategy = mitogen_linear` (cut processing time in half)
    * Ansible caching (little speed up)
    * SSH tunning (little speed up)

    Co-Authored-By: Mark Goddard <email address hidden>
    Closes-Bug: #1921538
    Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/802948

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/802949

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/803010

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/802948
Committed: https://opendev.org/openstack/kolla-ansible/commit/feb274b067d7b8e1da07307bb9f9d1c5172fae5b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit feb274b067d7b8e1da07307bb9f9d1c5172fae5b
Author: Rafael Weingärtner <email address hidden>
Date: Fri Mar 26 14:19:39 2021 -0300

    Make setup module arguments configurable

    Ansible facts can have a large impact on the performance of the Ansible
    control host. This patch introduces some control over which facts are
    gathered (kolla_ansible_setup_gather_subset) and which facts are stored
    (kolla_ansible_setup_filter). By default we do not change the default
    values of these arguments to the setup module. The flexibility of these
    arguments is limited, but they do provide enough for a large performance
    improvement in a typical moderate to large OpenStack cloud.

    In particular, the large complex dict fact for each interface has a
    large effect, and on an OpenStack controller or hypervisor there may be
    many virtual interfaces. We can use the kolla_ansible_setup_filter
    variable to help:

        kolla_ansible_setup_filter: 'ansible_[!qt]*'

    This causes Ansible to collect but not store facts matching that
    pattern, which includes the virtual interface facts. Currently we are
    not referencing other facts matching the pattern within Kolla Ansible.
    Note that including the 'ansible_' prefix causes meta facts module_setup
    and gather_subset to be filtered, but this seems to be the only way to
    get a good match on the interface facts. To work around this, we use
    ansible_facts rather than module_setup to detect whether facts exist in
    the cache.

    The exact improvement will vary, but has been reported to be as large as
    18x on systems with many virtual interfaces.

    For reference, here are some other tunings tried:

    * Increased the number of forks (great speedup depending of the size of
      the deployment)
    * Use `strategy = mitogen_linear` (cut processing time in half)
    * Ansible caching (little speed up)
    * SSH tunning (little speed up)

    Co-Authored-By: Mark Goddard <email address hidden>
    Closes-Bug: #1921538
    Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805
    (cherry picked from commit 15f2fdcd5dd9e63b477c549c2af990df8235ca90)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/802949
Committed: https://opendev.org/openstack/kolla-ansible/commit/da392e58bd4faf43dd29ec484ebf0c4fa81c3bdd
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit da392e58bd4faf43dd29ec484ebf0c4fa81c3bdd
Author: Rafael Weingärtner <email address hidden>
Date: Fri Mar 26 14:19:39 2021 -0300

    Make setup module arguments configurable

    Ansible facts can have a large impact on the performance of the Ansible
    control host. This patch introduces some control over which facts are
    gathered (kolla_ansible_setup_gather_subset) and which facts are stored
    (kolla_ansible_setup_filter). By default we do not change the default
    values of these arguments to the setup module. The flexibility of these
    arguments is limited, but they do provide enough for a large performance
    improvement in a typical moderate to large OpenStack cloud.

    In particular, the large complex dict fact for each interface has a
    large effect, and on an OpenStack controller or hypervisor there may be
    many virtual interfaces. We can use the kolla_ansible_setup_filter
    variable to help:

        kolla_ansible_setup_filter: 'ansible_[!qt]*'

    This causes Ansible to collect but not store facts matching that
    pattern, which includes the virtual interface facts. Currently we are
    not referencing other facts matching the pattern within Kolla Ansible.
    Note that including the 'ansible_' prefix causes meta facts module_setup
    and gather_subset to be filtered, but this seems to be the only way to
    get a good match on the interface facts. To work around this, we use
    ansible_facts rather than module_setup to detect whether facts exist in
    the cache.

    The exact improvement will vary, but has been reported to be as large as
    18x on systems with many virtual interfaces.

    For reference, here are some other tunings tried:

    * Increased the number of forks (great speedup depending of the size of
      the deployment)
    * Use `strategy = mitogen_linear` (cut processing time in half)
    * Ansible caching (little speed up)
    * SSH tunning (little speed up)

    Co-Authored-By: Mark Goddard <email address hidden>
    Closes-Bug: #1921538
    Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805
    (cherry picked from commit 15f2fdcd5dd9e63b477c549c2af990df8235ca90)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/803010
Committed: https://opendev.org/openstack/kolla-ansible/commit/a2ac590fcdec0d75cc8969d3515a83bd559457ab
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a2ac590fcdec0d75cc8969d3515a83bd559457ab
Author: Rafael Weingärtner <email address hidden>
Date: Fri Mar 26 14:19:39 2021 -0300

    Make setup module arguments configurable

    Ansible facts can have a large impact on the performance of the Ansible
    control host. This patch introduces some control over which facts are
    gathered (kolla_ansible_setup_gather_subset) and which facts are stored
    (kolla_ansible_setup_filter). By default we do not change the default
    values of these arguments to the setup module. The flexibility of these
    arguments is limited, but they do provide enough for a large performance
    improvement in a typical moderate to large OpenStack cloud.

    In particular, the large complex dict fact for each interface has a
    large effect, and on an OpenStack controller or hypervisor there may be
    many virtual interfaces. We can use the kolla_ansible_setup_filter
    variable to help:

        kolla_ansible_setup_filter: 'ansible_[!qt]*'

    This causes Ansible to collect but not store facts matching that
    pattern, which includes the virtual interface facts. Currently we are
    not referencing other facts matching the pattern within Kolla Ansible.
    Note that including the 'ansible_' prefix causes meta facts module_setup
    and gather_subset to be filtered, but this seems to be the only way to
    get a good match on the interface facts. To work around this, we use
    ansible_facts rather than module_setup to detect whether facts exist in
    the cache.

    The exact improvement will vary, but has been reported to be as large as
    18x on systems with many virtual interfaces.

    For reference, here are some other tunings tried:

    * Increased the number of forks (great speedup depending of the size of
      the deployment)
    * Use `strategy = mitogen_linear` (cut processing time in half)
    * Ansible caching (little speed up)
    * SSH tunning (little speed up)

    Co-Authored-By: Mark Goddard <email address hidden>
    Closes-Bug: #1921538
    Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805
    (cherry picked from commit 15f2fdcd5dd9e63b477c549c2af990df8235ca90)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 12.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 11.2.0 release.