neutron-server consumes all memcached connections

[Pierre Riteau]

This is a keystonemiddleware / neutron-server issue, but it potentially affects many Kolla deployments: https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

Reproduced on multiple train-centos8 deployments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/746966

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746966
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5a52d8e4a0c5d4c246deb8851ef893df63ee0847
Submitter: Zuul
Branch: master

commit 5a52d8e4a0c5d4c246deb8851ef893df63ee0847
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 19 17:56:30 2020 +0200

    Add workaround for keystonemiddleware/neutron memcached issue

    There is an issue where keystonemiddleware connections to memcached from
    neutron-server grow beyond configured values [1], eventually reaching
    the maximum number of connections accepted by memcached servers. Other
    services do not appear to be affected by this issue.

    A workaround is to use the advanced memcached pool. Despite its
    documentation claiming to only work with Python 2, it appears to work
    fine on Python 3.

    [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

    Change-Id: Ifbbc2022839cbc575848d830600241c61603c80b
    Closes-Bug: #1892210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/747658

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/747659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/747660

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747658
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=5d09ab68417705c37e94319f713abb20017e2559
Submitter: Zuul
Branch: stable/ussuri

commit 5d09ab68417705c37e94319f713abb20017e2559
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 19 17:56:30 2020 +0200

    Add workaround for keystonemiddleware/neutron memcached issue

    There is an issue where keystonemiddleware connections to memcached from
    neutron-server grow beyond configured values [1], eventually reaching
    the maximum number of connections accepted by memcached servers. Other
    services do not appear to be affected by this issue.

    A workaround is to use the advanced memcached pool. Despite its
    documentation claiming to only work with Python 2, it appears to work
    fine on Python 3.

    [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

    Change-Id: Ifbbc2022839cbc575848d830600241c61603c80b
    Closes-Bug: #1892210
    (cherry picked from commit 5a52d8e4a0c5d4c246deb8851ef893df63ee0847)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747660
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=7b512d633d59d8d35db785bfa3d7e61dd202d399
Submitter: Zuul
Branch: stable/stein

commit 7b512d633d59d8d35db785bfa3d7e61dd202d399
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 19 17:56:30 2020 +0200

    Add workaround for keystonemiddleware/neutron memcached issue

    There is an issue where keystonemiddleware connections to memcached from
    neutron-server grow beyond configured values [1], eventually reaching
    the maximum number of connections accepted by memcached servers. Other
    services do not appear to be affected by this issue.

    A workaround is to use the advanced memcached pool. Despite its
    documentation claiming to only work with Python 2, it appears to work
    fine on Python 3.

    [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

    Change-Id: Ifbbc2022839cbc575848d830600241c61603c80b
    Closes-Bug: #1892210
    (cherry picked from commit 5a52d8e4a0c5d4c246deb8851ef893df63ee0847)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747659
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=dcc74b45e18c56d22a66ee884f84ae31c0048622
Submitter: Zuul
Branch: stable/train

commit dcc74b45e18c56d22a66ee884f84ae31c0048622
Author: Pierre Riteau <email address hidden>
Date: Wed Aug 19 17:56:30 2020 +0200

    Add workaround for keystonemiddleware/neutron memcached issue

    There is an issue where keystonemiddleware connections to memcached from
    neutron-server grow beyond configured values [1], eventually reaching
    the maximum number of connections accepted by memcached servers. Other
    services do not appear to be affected by this issue.

    A workaround is to use the advanced memcached pool. Despite its
    documentation claiming to only work with Python 2, it appears to work
    fine on Python 3.

    [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659

    Change-Id: Ifbbc2022839cbc575848d830600241c61603c80b
    Closes-Bug: #1892210
    (cherry picked from commit 5a52d8e4a0c5d4c246deb8851ef893df63ee0847)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.3.0 release.