Fail to establish connection to keystone in case if master node has big numbers of cpu

[Andrey Grebennikov]

MOS8

Fuel node has 48 vcpus. Keystone docker container configures Keystone to have the number of workers equal to the doubled number of vcpus (in my case is is 48+48=96).
At the same time Postgres DB allows 100 connections by default.
Eventually it comes to the point when neither of Fuel CLI commands work since it cannot connect to Postgres, as well as OSTF server cannot establish new connections because of the same reason.

It is necessary find a way to limit the number of workers for the Keystone on the Fuel node.

[Tatyanka (tatyana-leontovich)]

ostf does not configure keystone

[Andrey Grebennikov (agrebennikov)]

FYI - I believe MOS8 MU should also include it - for real customers it affects literaly every deployment (if Fuel is deployed to baremetal node with characteristics similar to the cloud nodes).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/347857

[Alex Schultz (alex-schultz)]

Invalid for newton as we've switched to using apache and there is already a limit in place for the maximum number of wsgi workers

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347857
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=7bd587895bf3c6298bb8bda438a7b931d0bf0ca7
Submitter: Jenkins
Branch: stable/mitaka

commit 7bd587895bf3c6298bb8bda438a7b931d0bf0ca7
Author: Alex Schultz <email address hidden>
Date: Wed Jul 27 08:05:19 2016 -0600

    Set upper limit on keystone workers

    This change sets an upper limit on the keystone workers to 16. It will
    use the processor count to determine the correct number of workers for
    smaller environments so it uses the smaller of the values between
    processorcount or 16. This change sets the number of admin workers and
    public workers to the lesser of these two values for a maximum of 32
    workers total.

    Change-Id: Ibbc2704441806776ded62e83b26b043188ad1a42
    Closes-Bug: #1602434

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/366147

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/366697

[Nastya Urlapova (aurlapova)]

Marked as "Verified", because we don't have server with 48CPUs, we cannot do a proper verification.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366697
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=90f41ffa91bdbccb3218fe7824633bc3c8b5864e
Submitter: Jenkins
Branch: stable/7.0

commit 90f41ffa91bdbccb3218fe7824633bc3c8b5864e
Author: Anton Chevychalov <email address hidden>
Date: Wed Sep 7 14:42:07 2016 +0300

    Set upper limit on keystone workers

    This change sets an upper limit on the keystone workers to 16. It will
    use the processor count to determine the correct number of workers for
    smaller environments so it uses the smaller of the values between
    processorcount or 16. This change sets the number of admin workers and
    public workers to the lesser of these two values for a maximum of 32
    workers total.

    Backport of 7bd587895bf3c6298bb8bda438a7b931d0bf0ca7

    Change-Id: Ibbc2704441806776ded62e83b26b043188ad1a42
    Closes-Bug: #1602434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366147
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=05c26f6962228eeb6282e325c4bfc237a0906923
Submitter: Jenkins
Branch: stable/8.0

commit 05c26f6962228eeb6282e325c4bfc237a0906923
Author: Anton Chevychalov <email address hidden>
Date: Tue Sep 6 16:50:05 2016 +0300

    Set upper limit on keystone workers

    This change sets an upper limit on the keystone workers to 16. It will
    use the processor count to determine the correct number of workers for
    smaller environments so it uses the smaller of the values between
    processorcount or 16. This change sets the number of admin workers and
    public workers to the lesser of these two values for a maximum of 32
    workers total.

    Backport of 7bd587895bf3c6298bb8bda438a7b931d0bf0ca7

    Change-Id: Ibbc2704441806776ded62e83b26b043188ad1a42
    Closes-Bug: #1602434

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 + MU6 updates.

Actual results:
1) If fuel node has 18 CPUs:
Before updates:
[root@nailgun ~]# dockerctl shell keystone
[root@nailgun ~]# cat /etc/keystone/keystone.conf | grep workers
public_workers=18
admin_workers=18

After updates:
[root@nailgun ~]# dockerctl shell keystone
[root@nailgun ~]# cat /etc/keystone/keystone.conf | grep workers
public_workers=16
admin_workers=16

2) If fuel node has 1 CPU or 5 CPUs, for example, then before and after updates the count of admin and public workers 2+2 or 5+5 respectively.

So, after updates max possible number of admin/public workers is 16, total count is 32.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421376

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421419

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU4 updates.

Actual results:
1) If fuel node has 18 CPUs:
Before updates:
[root@nailgun ~]# dockerctl shell keystone
[root@nailgun ~]# cat /etc/keystone/keystone.conf | grep workers
public_workers=18
admin_workers=18

After updates:
[root@nailgun ~]# dockerctl shell keystone
[root@nailgun ~]# cat /etc/keystone/keystone.conf | grep workers
public_workers=16
admin_workers=16

2) If fuel node has 1 CPU or 5 CPUs, for example, then before and after updates the count of admin and public workers 2+2 or 5+5 respectively.

So, after updates max possible number of admin/public workers is 16, total count is 32.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/421376
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/421419
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.