Set [keystone_authtoken] memcache_use_advanced_pool to avoid bursting connections to memcached

[Takashi Kajinami]

Description
===========

This issue was initially reported in the following downstream bug.

https://bugzilla.redhat.com/show_bug.cgi?id=1893205

We noticed sometimes memcached has huge number of connections established and hit max connection limit(8192 by default).

Looking at all connections, it turned out specific services like neutron have huge number (thousands) of connections with memcached, which is triggered by keystonemiddleware.

While we'd need to identify the root cause in keystonemiddleware, so far setting the following parameter works as a valid workaround and we no longer observe bursting connections with this setting.

~~~
[keystone_authtoken]
memcache_use_advanced_pool = True
~~~

This bug is to track an effort to add a tht parameter to set that keystone_authtoken parameter.

Steps to reproduce
==================
1. Deploy overcloud
2. Execute API requests periodically
3. Monitor number of connections established with memcached

Expected result
===============
Number of connections stays at some point

Actual result
=============
Number of connections keeps growing

Environment
===========
The issue was initially found in stable/train

Logs & Configs
==============
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795010

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795497

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795010
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/09bcacd25a7217ce66eec10ccfda4847e6d3d87a
Submitter: "Zuul (22348)"
Branch: master

commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a
Author: Takashi Kajinami <email address hidden>
Date: Mon Jun 7 10:31:32 2021 +0900

    Add support for keystone_authtoken/memcache_use_advanced_pool

    This change introduces a single parameter, MemcacheUseAdvancedPool,
    to enable usage of advanced connection pool in keystone middleware.
    This is useful to avoid bursting connection to memcached.

    Note that the default value of memcached_use_advanced_pool was changed
    from false to true during Xena cycle[1], so this parameter is no longer
    required in master. However the change in keystonemiddleware will
    never be backported. This change is created so that we can switch to
    advanced pool even in older releases.

    [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/773939

    Closes-Bug: #1931047
    Change-Id: I2887249af44ccfdae1592dd9120d3366fa059876

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796248

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796703

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797088

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796248
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ef18f2515852b17064d8fdd4d9ad65fc42492135
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ef18f2515852b17064d8fdd4d9ad65fc42492135
Author: Takashi Kajinami <email address hidden>
Date: Mon Jun 7 10:31:32 2021 +0900

    Add support for keystone_authtoken/memcache_use_advanced_pool

    This change introduces a single parameter, MemcacheUseAdvancedPool,
    to enable usage of advanced connection pool in keystone middleware.
    This is useful to avoid bursting connection to memcached.

    Note that the default value of memcached_use_advanced_pool was changed
    from false to true during Xena cycle[1], so this parameter is no longer
    required in master. However the change in keystonemiddleware will
    never be backported. This change is created so that we can switch to
    advanced pool even in older releases.

    [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/773939

    Closes-Bug: #1931047
    Change-Id: I2887249af44ccfdae1592dd9120d3366fa059876
    (cherry picked from commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796703
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/89fde273eaf69bc32d79221883c7abf657ebbda9
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 89fde273eaf69bc32d79221883c7abf657ebbda9
Author: Takashi Kajinami <email address hidden>
Date: Mon Jun 7 10:31:32 2021 +0900

    Add support for keystone_authtoken/memcache_use_advanced_pool

    This change introduces a single parameter, MemcacheUseAdvancedPool,
    to enable usage of advanced connection pool in keystone middleware.
    This is useful to avoid bursting connection to memcached.

    Note that the default value of memcached_use_advanced_pool was changed
    from false to true during Xena cycle[1], so this parameter is no longer
    required in master. However the change in keystonemiddleware will
    never be backported. This change is created so that we can switch to
    advanced pool even in older releases.

    [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/773939

    (To Victoria)
    Conflicts:
            deployment/aodh/aodh-api-container-puppet.yaml
            deployment/glance/glance-api-container-puppet.yaml
            deployment/gnocchi/gnocchi-api-container-puppet.yaml
            deployment/manila/manila-api-container-puppet.yaml
            deployment/nova/nova-api-container-puppet.yaml

    Closes-Bug: #1931047
    Change-Id: I2887249af44ccfdae1592dd9120d3366fa059876
    (cherry picked from commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a)
    (cherry picked from commit 9a89b9add62d1d6251285a091e96e98f8157d1bd)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/800024

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/800024
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/59599a86d7cd69240195f4455d8fe5e76c9ace51
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 59599a86d7cd69240195f4455d8fe5e76c9ace51
Author: Takashi Kajinami <email address hidden>
Date: Thu Jul 8 21:19:54 2021 +0900

    Enable keystone_authtoken/memcach_use_advanced_pool for Sahara

    Sahara was removed during Wallaby cycle and because of that it was not
    updated when commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a was
    backported from Wallaby to Victoria.
    This change updates the Sahara templates to have consistent definition
    with the other services.

    Related-Bug: #1931047
    Change-Id: Idb9a209b2651c1ff03c3ccaa14f4976a9f267e99

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797088
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/bef9856ab1fcc7bf6e358f03c775a86705b3b985
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit bef9856ab1fcc7bf6e358f03c775a86705b3b985
Author: Takashi Kajinami <email address hidden>
Date: Mon Jun 7 10:31:32 2021 +0900

    Add support for keystone_authtoken/memcache_use_advanced_pool

    This change introduces a single parameter, MemcacheUseAdvancedPool,
    to enable usage of advanced connection pool in keystone middleware.
    This is useful to avoid bursting connection to memcached.

    Note that the default value of memcached_use_advanced_pool was changed
    from false to true during Xena cycle[1], so this parameter is no longer
    required in master. However the change in keystonemiddleware will
    never be backported. This change is created so that we can switch to
    advanced pool even in older releases.

    [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/773939

    (To Victoria)
    Conflicts:
            deployment/aodh/aodh-api-container-puppet.yaml
            deployment/glance/glance-api-container-puppet.yaml
            deployment/gnocchi/gnocchi-api-container-puppet.yaml
            deployment/manila/manila-api-container-puppet.yaml
            deployment/nova/nova-api-container-puppet.yaml

    Closes-Bug: #1931047
    Change-Id: I2887249af44ccfdae1592dd9120d3366fa059876
    (cherry picked from commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a)
    (cherry picked from commit 9a89b9add62d1d6251285a091e96e98f8157d1bd)
    (cherry picked from commit 89fde273eaf69bc32d79221883c7abf657ebbda9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/795497
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fb7df92436236c3cf82a5953492b2f8962259f69
Submitter: "Zuul (22348)"
Branch: stable/train

commit fb7df92436236c3cf82a5953492b2f8962259f69
Author: Takashi Kajinami <email address hidden>
Date: Mon Jun 7 10:31:32 2021 +0900

    Add support for keystone_authtoken/memcache_use_advanced_pool

    This change introduces a single parameter, MemcacheUseAdvancedPool,
    to enable usage of advanced connection pool in keystone middleware.
    This is useful to avoid bursting connection to memcached.

    Note that the default value of memcached_use_advanced_pool was changed
    from false to true during Xena cycle[1], so this parameter is no longer
    required in master. However the change in keystonemiddleware will
    never be backported. This change is created so that we can switch to
    advanced pool even in older releases.

    [1] https://review.opendev.org/c/openstack/keystonemiddleware/+/773939

    (To Victoria)
    Conflicts:
            deployment/aodh/aodh-api-container-puppet.yaml
            deployment/glance/glance-api-container-puppet.yaml
            deployment/gnocchi/gnocchi-api-container-puppet.yaml
            deployment/manila/manila-api-container-puppet.yaml
            deployment/nova/nova-api-container-puppet.yaml

    (To Train)
    Conflicts:
            deployment/aodh/aodh-base.yaml
            deployment/barbican/barbican-api-container-puppet.yaml
            deployment/cinder/cinder-api-container-puppet.yaml
            deployment/experimental/designate/designate-api-container-puppet.yaml
            deployment/glance/glance-api-container-puppet.yaml
            deployment/gnocchi/gnocchi-api-container-puppet.yaml
            deployment/heat/heat-base-puppet.yaml
            deployment/ironic/ironic-api-container-puppet.yaml
            deployment/manila/manila-api-container-puppet.yaml
            deployment/mistral/mistral-base.yaml
            deployment/neutron/neutron-api-container-puppet.yaml
            deployment/nova/nova-api-container-puppet.yaml
            deployment/nova/nova-metadata-container-puppet.yaml
            deployment/nova/novajoin-container-puppet.yaml
            deployment/octavia/octavia-api-container-puppet.yaml
            deployment/placement/placement-api-container-puppet.yaml
            deployment/zaqar/zaqar-container-puppet.yaml

    Also fixed Panko which is no longer supported since Ussuri.

    Closes-Bug: #1931047
    Change-Id: I2887249af44ccfdae1592dd9120d3366fa059876
    (cherry picked from commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a)
    (cherry picked from commit 9a89b9add62d1d6251285a091e96e98f8157d1bd)
    (cherry picked from commit 89fde273eaf69bc32d79221883c7abf657ebbda9)
    (cherry picked from commit bef9856ab1fcc7bf6e358f03c775a86705b3b985)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.5 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/806908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/806908
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3b5628babca75abb59f0a381eebfd0a03dfb1160
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3b5628babca75abb59f0a381eebfd0a03dfb1160
Author: Takashi Kajinami <email address hidden>
Date: Thu Jul 8 21:19:54 2021 +0900

    Enable keystone_authtoken/memcach_use_advanced_pool for Sahara

    Sahara was removed during Wallaby cycle and because of that it was not
    updated when commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a was
    backported from Wallaby to Victoria.
    This change updates the Sahara templates to have consistent definition
    with the other services.

    Related-Bug: #1931047
    Change-Id: Idb9a209b2651c1ff03c3ccaa14f4976a9f267e99
    (cherry picked from commit 59599a86d7cd69240195f4455d8fe5e76c9ace51)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/812640

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/812640
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/780cd556a7ba36d9f8e102debe36cb756d6fde24
Submitter: "Zuul (22348)"
Branch: stable/train

commit 780cd556a7ba36d9f8e102debe36cb756d6fde24
Author: Takashi Kajinami <email address hidden>
Date: Thu Jul 8 21:19:54 2021 +0900

    Enable keystone_authtoken/memcach_use_advanced_pool for Sahara

    Sahara was removed during Wallaby cycle and because of that it was not
    updated when commit 09bcacd25a7217ce66eec10ccfda4847e6d3d87a was
    backported from Wallaby to Victoria.
    This change updates the Sahara templates to have consistent definition
    with the other services.

    Conflicts:
            deployment/sahara/sahara-base.yaml

    Related-Bug: #1931047
    Change-Id: Idb9a209b2651c1ff03c3ccaa14f4976a9f267e99
    (cherry picked from commit 59599a86d7cd69240195f4455d8fe5e76c9ace51)
    (cherry picked from commit 3b5628babca75abb59f0a381eebfd0a03dfb1160)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.