ceph-ansible fork count should be a parameter

Bug #1738276 reported by John Fulton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
John Fulton

Bug Description

Starting in Pike TripleO sets the ansible fork count to maximize the number of concurrent ceph-ansible processes [0] by setting it equal to the number of nodes needing to be configured up to 100. However, if the undercloud doesn't have enough memory, then this setting cause the undercloud to run out of RAM [1]. The user should be able to easily override this parameter and not use the formula.

[0] https://github.com/openstack/tripleo-common/blob/fa0b9f52080580b7408dc6f5f2da6fc1dc07d500/workbooks/ceph-ansible.yaml#L25

[1]
2017-12-13 04:05:53,887 p=7259 u=mistral | ERROR! Unexpected Exception, this is probably a bug: [Errno 12] Cannot allocate memory │····················
2017-12-13 04:05:53,893 p=7259 u=mistral | to see the full traceback, use -vvv │····················
2017-12-13 04:05:58,541 p=7259 u=mistral | the full traceback was: │····················
                                                                                                                                                                                              │····················
Traceback (most recent call last): │····················
  File "/bin/ansible-playbook", line 106, in <module> │····················
    exit_code = cli.run() │····················
  File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 130, in run │····················
    results = pbex.run() │····················
  File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 154, in run │····················
    result = self._tqm.run(play=play) │····················
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 302, in run │····················
    play_return = strategy.run(iterator, play_context) │····················
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 277, in run │····················
    self._queue_task(host, task, task_vars, play_context) │····················
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/__init__.py", line 222, in _queue_task │····················
    worker_prc.start() │····················
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start │····················
    self._popen = Popen(self) │····················
  File "/usr/lib64/python2.7/multiprocessing/forking.py", line 121, in __init__ │····················
    self.pid = os.fork() │····················
OSError: [Errno 12] Cannot allocate memory

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/528124

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/528125

Changed in tripleo:
assignee: John Fulton (jfulton-org) → Alex Schultz (alex-schultz)
Changed in tripleo:
assignee: Alex Schultz (alex-schultz) → John Fulton (jfulton-org)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/528124
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=97185448b307cea98a2e2f11337df4950e76b95d
Submitter: Zuul
Branch: master

commit 97185448b307cea98a2e2f11337df4950e76b95d
Author: John Fulton <email address hidden>
Date: Thu Dec 14 18:20:24 2017 -0500

    Parameterize Ansible environment vars in Mistral Workflow

    Add ansible_env_variables which a user may use to override
    any Ansible environment variable. This new parameter is
    then merged with the default env variables as set in
    ansible_extra_env_variables.

    Also adds DEFAULT_FORKS to the default env variables
    with a new default of 50. Ansible automatically limits
    the number of forks to the number of possible hosts at
    runtime so it is not necessary to compute the fork count.

    Change-Id: I5d69af146ca6ca8b3d5f78445cd1b47828daa955
    Partial-Bug: 1738276

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/528125
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ba2169d0a39fb2b14f0a261055e8a41270cfa4e1
Submitter: Zuul
Branch: master

commit ba2169d0a39fb2b14f0a261055e8a41270cfa4e1
Author: John Fulton <email address hidden>
Date: Thu Dec 14 18:21:14 2017 -0500

    Parameterize ceph-ansible environment variables

    Add CephAnsibleEnvironmentVariables which allows a user to
    override any Ansible environment variable.

    Depends-On: I5d69af146ca6ca8b3d5f78445cd1b47828daa955
    Change-Id: Ic731c8f0c988c485c5b3448182a568b8514cab0a
    Closes-Bug: 1738276

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/535291

tags: added: pike-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/535292

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/pike)

Reviewed: https://review.openstack.org/535291
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=dd8098e4d953eec210e7edb69cd8e494c0080238
Submitter: Zuul
Branch: stable/pike

commit dd8098e4d953eec210e7edb69cd8e494c0080238
Author: John Fulton <email address hidden>
Date: Thu Dec 14 18:20:24 2017 -0500

    Parameterize Ansible environment vars in Mistral Workflow

    Add ansible_env_variables which a user may use to override
    any Ansible environment variable. This new parameter is
    then merged with the default env variables as set in
    ansible_extra_env_variables.

    Also adds DEFAULT_FORKS to the default env variables
    with a new default of 50. Ansible automatically limits
    the number of forks to the number of possible hosts at
    runtime so it is not necessary to compute the fork count.

    Change-Id: I5d69af146ca6ca8b3d5f78445cd1b47828daa955
    Partial-Bug: 1738276
    (cherry picked from commit 97185448b307cea98a2e2f11337df4950e76b95d)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.0.0b3

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/pike)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/535292
Reason: Gate resets is causing issue to land a critical patch (538346) - I need to clear the gate until 538346 is merged. Please do not restore this patch, I'll take care of it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/pike)

Reviewed: https://review.openstack.org/535292
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=75a2d519cbecee850b18f802df51c52d8c6ca65e
Submitter: Zuul
Branch: stable/pike

commit 75a2d519cbecee850b18f802df51c52d8c6ca65e
Author: John Fulton <email address hidden>
Date: Thu Dec 14 18:21:14 2017 -0500

    Parameterize ceph-ansible environment variables

    Add CephAnsibleEnvironmentVariables which allows a user to
    override any Ansible environment variable.

    Depends-On: I5d69af146ca6ca8b3d5f78445cd1b47828daa955
    Change-Id: Ic731c8f0c988c485c5b3448182a568b8514cab0a
    Closes-Bug: 1738276
    (cherry picked from commit ba2169d0a39fb2b14f0a261055e8a41270cfa4e1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.9

This issue was fixed in the openstack/tripleo-heat-templates 7.0.9 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.