high memory usage over time

[James Page]

On our QA cloud, we observed that the gnocchi api processes under Apache2 where consuming a very large amount of the RAM on the machine (100G) - this appears to be due to the wsgi configuration in Apache configuring 10 threads as well as 4 processes.

https://bugzilla.redhat.com/show_bug.cgi?id=1596591

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/621137

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621137
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=5c165a01634f67566bcb9dd8cd15fe402a7cdeec
Submitter: Zuul
Branch: master

commit 5c165a01634f67566bcb9dd8cd15fe402a7cdeec
Author: James Page <email address hidden>
Date: Fri Nov 30 09:57:40 2018 +0000

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Gnocchi WSGI application, avoiding high memory consumption
    over time under load.

    Change-Id: I3d92c6997ec2129ab6ad161caac941f19cd6fbfe
    Closes-Bug: 1806008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/621532

[Frode Nordahl (fnordahl)]

Fix proposed to branch: master
Review: https://review.openstack.org/#/c/623136/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621532
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=ef979d10d27ad9ee3fe34c1ec453ed2b555c20c8
Submitter: Zuul
Branch: stable/18.11

commit ef979d10d27ad9ee3fe34c1ec453ed2b555c20c8
Author: James Page <email address hidden>
Date: Fri Nov 30 09:57:40 2018 +0000

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Gnocchi WSGI application, avoiding high memory consumption
    over time under load.

    Change-Id: I3d92c6997ec2129ab6ad161caac941f19cd6fbfe
    Closes-Bug: 1806008
    (cherry picked from commit 5c165a01634f67566bcb9dd8cd15fe402a7cdeec)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/623136
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=1a577cea6ef280d4ef6ed95e8e0861f422dcdb01
Submitter: Zuul
Branch: master

commit 1a577cea6ef280d4ef6ed95e8e0861f422dcdb01
Author: Frode Nordahl <email address hidden>
Date: Thu Dec 6 05:42:39 2018 +0100

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Octavia WSGI application, avoiding high memory consumption
    over time under load.

    Change-Id: Ia4b0e6b6b364dc3c9cd2fa4cdfa93cd2766c4d6a
    Closes-Bug: #1806008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/623527

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/623527
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=3e6bcfb18cb4832289b4d2b84d1a06dedfe04439
Submitter: Zuul
Branch: stable/18.11

commit 3e6bcfb18cb4832289b4d2b84d1a06dedfe04439
Author: Frode Nordahl <email address hidden>
Date: Thu Dec 6 05:42:39 2018 +0100

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Octavia WSGI application, avoiding high memory consumption
    over time under load.

    Change-Id: Ia4b0e6b6b364dc3c9cd2fa4cdfa93cd2766c4d6a
    Closes-Bug: #1806008
    (cherry picked from commit 1a577cea6ef280d4ef6ed95e8e0861f422dcdb01)

[Corey Bryant (corey.bryant)]

I don't think this was intended but it looks like this was only fixed for gnocchi mitaka. There's also a gnocchi-api.conf in the newton directory:

.
├── api-paste.ini
├── ceph.conf
├── gnocchi-api.conf
├── gnocchi.conf
├── newton
│   └── gnocchi-api.conf
├── ocata
│   ├── api-paste.ini
│   ├── gnocchi-nginx.conf
│   └── nginx.conf
└── queens
    ├── api-paste.ini
    └── gnocchi.conf

[Giuseppe Petralia (peppepetra)]

As mentioned in the comment before, the bug is still valid when charm is using the template in the newton folder (i.e. cloud:xenial-queens).

https://github.com/openstack/charm-gnocchi/blob/master/src/templates/newton/gnocchi-api.conf

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/730347

[James Page (james-page)]

Working on the missed template as I missed it the first time around.

[James Page (james-page)]

https://review.opendev.org/730347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730347
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=d684a6e73a7e6ba3809b591911c10b0318c2bcfd
Submitter: Zuul
Branch: master

commit d684a6e73a7e6ba3809b591911c10b0318c2bcfd
Author: James Page <email address hidden>
Date: Fri May 22 15:31:38 2020 +0100

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Gnocchi WSGI application, avoiding high memory consumption
    over time under load.

    A previous version of this change missed the newton specific version
    of the apache configuration for gnocchi api.

    Change-Id: Idb355dc5f5fa404140819f4b28940e9bbc1693e6
    Closes-Bug: 1806008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.05
Review: https://review.opendev.org/730721

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/730721
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=2beab5b7cdc3c9bd46ad34756f35912463a12893
Submitter: Zuul
Branch: stable/20.05

commit 2beab5b7cdc3c9bd46ad34756f35912463a12893
Author: James Page <email address hidden>
Date: Fri May 22 15:31:38 2020 +0100

    Drop wsgi thread configuration to a single thread

    We should increase concurrency using processes rather than threads
    for the Gnocchi WSGI application, avoiding high memory consumption
    over time under load.

    A previous version of this change missed the newton specific version
    of the apache configuration for gnocchi api.

    Change-Id: Idb355dc5f5fa404140819f4b28940e9bbc1693e6
    Closes-Bug: 1806008
    (cherry picked from commit d684a6e73a7e6ba3809b591911c10b0318c2bcfd)