mysql process consuming 400% CPU

[Nagasai Vinaykumar Kapalavai]

Description of problem:
On all three controllers, the mysql process is consuming around 4000% CPU.

Version-Release number of selected component (if applicable):

How reproducible:
Hard to reproduce outside of customer environment
Observed in ths OSP13,14,15

Steps to Reproduce:
1. Deploy overcloud with or without composable roles
2. Check CPU utilization of mysql process
3.

Actual results:
Very high CPU utilisation. Even after restarting galera from pcs we're seeing 400% CPU utilization. No iowait observed

Expected results:
Much lower CPU utilization

This occurs when the panko events grow indefinitely and as a result mysql process consumes 400% cpu to generate query results. The current solution to this is running the panko_api_cron in a seperate container.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664048
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7b1e923ede7a2c522e26c5dd9e9ec7fe9da026d7
Submitter: Zuul
Branch: stable/queens

commit 7b1e923ede7a2c522e26c5dd9e9ec7fe9da026d7
Author: Vinay Kapalavai <email address hidden>
Date: Fri Jun 7 15:11:43 2019 -0400

    Add panko_api_cron container

    Currently panko expirer isn't running in a separate container,
    which results in events piling up in the configured database.
    This change deploys panko_api_cron as a container and
    periodically deletes events from db

    Closes-Bug: #1832274
    Change-Id: I4c941048a3561af9458208fe86e1b86156eb7ebc
    (cherry picked from commit 66a638a8df8fa5b7c9bb92eb1e9da9d09c1fb447)
    (cherry picked from commit 9e876dfe0d743d51283a3462d03991180d3ace51)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664047
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4354960827e9a260ac9e72cf402edf2649ab35cd
Submitter: Zuul
Branch: stable/rocky

commit 4354960827e9a260ac9e72cf402edf2649ab35cd
Author: Vinay Kapalavai <email address hidden>
Date: Fri Jun 7 15:11:43 2019 -0400

    Add panko_api_cron container

    Currently panko expirer isn't running in a separate container,
    which results in events piling up in the configured database.
    This change deploys panko_api_cron as a container and
    periodically deletes events from db

    Closes-Bug: #1832274
    Change-Id: I4c941048a3561af9458208fe86e1b86156eb7ebc
    (cherry picked from commit 66a638a8df8fa5b7c9bb92eb1e9da9d09c1fb447)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664040
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c2ceefcd5804ae3cdf6db138da2f89e3d2fa986d
Submitter: Zuul
Branch: stable/stein

commit c2ceefcd5804ae3cdf6db138da2f89e3d2fa986d
Author: Vinay Kapalavai <email address hidden>
Date: Fri Jun 7 15:11:43 2019 -0400

    Add panko_api_cron container

    Currently panko expirer isn't running in a separate container,
    which results in events piling up in the configured database.
    This change deploys panko_api_cron as a container and
    periodically deletes events from db

    Closes-Bug: #1832274
    Change-Id: I4c941048a3561af9458208fe86e1b86156eb7ebc
    (cherry picked from commit 66a638a8df8fa5b7c9bb92eb1e9da9d09c1fb447)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.