Sysinv-api crashed during 250 subcloud deployment

[Iago Filipe]

Brief Description
Many deploy-prep-failed errors were generated during the 250 subcloud deployment test which eventually took down both sysinv-api and dcmanager-manager.

Severity
Major

Steps to Reproduce
Deploy a batch of 250 virtual subclouds

Expected Behavior
This is an explorative test to identify the bottlenecks of 250 parallel subcloud deployments (without remote install). I'd expect some deployment failures but I did not expect a service crash.

Actual Behavior
Sysinv-api restarts after more or less 150 route creation requests in parallel.

Reproducibility
100% reproducible

System Configuration
Distributed Cloud
Starlingx Master
BUILD_DATE="2022-04-02 20:07:42 -0400"

Last Pass
N/A

Timestamp/Logs

Alarms

Test Activity
Developer Testing

Workaround

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/842599

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/842604

[Ghada Khalil (gkhalil)]

screening: lowering the priority given this is a scalability issue. This doesn't hold up the stx.7.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/842599
Committed: https://opendev.org/starlingx/config/commit/dc7c3c93800a17e4031444b91880f84b0a33fb28
Submitter: "Zuul (22348)"
Branch: master

commit dc7c3c93800a17e4031444b91880f84b0a33fb28
Author: Iago Estrela <email address hidden>
Date: Thu May 19 10:49:20 2022 -0300

    Fix sysinv-api crash with 250 parallel requests

    Sysinv WSGI server is crashing after receiving 250 requests in parallel
    in route creation endpoint due to limited number of threads to handle
    requests.

    Closes-Bug: 1974194

    Test plan:
    PASS: Hit route creation endpoint 250 times in parallel and verify
          WSGI Server didn't restart.

    Signed-off-by: Iago Estrela <email address hidden>
    Change-Id: I89012d7f8c7693cd3dc078d9f67ddffb4308e254

[OpenStack Infra (hudson-openstack)]

Change abandoned by "John Kung <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/842604
Reason: Abandon review, reason as per prior comment