RPM catalogs wasn't retrieved from masternode under high load

[Aleksandr Shaposhnikov]

Problem: during 100 nodes deployment not all the nodes was able to retrieve RPM catalog from MOS/FUEL masternode.
Screenshot from one of the nodes is in attachments.
Diagnostic snapshot is also in attachments.

[Matthew Mosesohn (raytrac3r)]

We could see if it's possible to modify yum config in %pre section of kickstart to set timeout=240. The default is 30. I know that if the server doesn't send any data before timeout, the connection is considered bad and results in an error.

[Sergii Golovatiuk (sgolovatiuk)]

One option we can tune nginx on master node. At the moment it has pretty standard options. They are not set for scale and parralel downloads.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130523

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130523
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5c6320638e993e37f4914185ed693c2b80be1438
Submitter: Jenkins
Branch: master

commit 5c6320638e993e37f4914185ed693c2b80be1438
Author: Matthew Mosesohn <email address hidden>
Date: Thu Oct 23 17:01:23 2014 +0400

    Add timeout for anaconda yum conf

    Raises timeout from 30s to 240s so that installs
    do not fail on yum operations during install under
    heavy load.

    Change-Id: If2a67b15beb6371979f1638d2c49ef935bfbed50
    Partial-Bug: #1384510

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131672
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9ea5104a432b0303f35606a3d7dc880f48f76d2f
Submitter: Jenkins
Branch: master

commit 9ea5104a432b0303f35606a3d7dc880f48f76d2f
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Oct 29 09:56:02 2014 +0100

    Set nginx default settings for master node

    - increase number of workers from 2 to half available CPUs.
    - enable epoll by default
    - enable tcpnopush
    - enable tcpnodelay
    - disable server_token

    These settings allow to bootstrap many nodes in parallel allowing
    anaconda and debian installer to get all files without timeout.

    Partial-Bug: 1384510
    Partial-Bug: 1349360

    Change-Id: I44054cf2afa1fa5a70ba064f4957b964947b2b2c
    Implements: blueprint 100-nodes-support
    Signed-off-by: Sergii Golovatiuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/134223

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134223
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9ee34175ffd0070fd5041ef3bc052d452f009654
Submitter: Jenkins
Branch: stable/5.1

commit 9ee34175ffd0070fd5041ef3bc052d452f009654
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Oct 29 09:56:02 2014 +0100

    Set nginx default settings for master node

    - increase number of workers from 2 to half available CPUs.
    - enable epoll by default
    - enable tcpnopush
    - enable tcpnodelay
    - disable server_token

    These settings allow to bootstrap many nodes in parallel allowing
    anaconda and debian installer to get all files without timeout.

    Partial-Bug: 1384510
    Partial-Bug: 1349360

    Change-Id: I44054cf2afa1fa5a70ba064f4957b964947b2b2c
    Implements: blueprint 100-nodes-support
    Signed-off-by: Sergii Golovatiuk <email address hidden>