Mistral's eventlet monkeypatching breaks git-python

[Emilien Macchi]

Originally posted here:
https://bugzilla.redhat.com/show_bug.cgi?id=1685621

Description of problem:
When deploying a TripleO Overcloud, the "openstack overcloud" commands uses mistral to signal progress on the deployment, and fails:

Enabling ssh admin - COMPLETE.
Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception [action_ex_id=23ec3e50-9a0f-448c-8f86-5f7553357fe3, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{'container': 'overcloud', 'container_config': 'overclou
d-config'}']
 module 'select' has no attribute 'poll'
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 29, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 945, in take_action
    verbosity=self.app_args.verbose_level)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 323, in config_download
    raise exceptions.DeploymentError("Overcloud configuration failed.")
tripleoclient.exceptions.DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.
Full error:
2019-03-02 03:43:20.127 1 WARNING mistral.executors.default_executor [req-0568807f-eb53-4a31-a05b-afcf0631330b 07085b77ed23481ab53cc174dfb53f08 ec0093095eae49b1a5e67d54894e2b0d - default default] The action raised an exception [action_ex_
id=23ec3e50-9a0f-448c-8f86-5f7553357fe3, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{'container': 'overcloud', 'container_config': 'overcloud-config'}']
 module 'select' has no attribute 'poll': AttributeError: module 'select' has no attribute 'poll'
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/tripleo_common/actions/config.py", line 87, in run
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor commit_message=message)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 492, in download_config
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor git_repo = self.initialize_git_repo(config_dir)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 140, in initialize_git_repo
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor repo = git.Repo.init(dirname)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/git/repo/base.py", line 764, in init
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor git.init(**kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/git/cmd.py", line 440, in <lambda>
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/git/cmd.py", line 834, in _call_process
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor return self.execute(make_call(), **_kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/git/cmd.py", line 587, in execute
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor stdout_value, stderr_value = proc.communicate()
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor stdout, stderr = self._communicate(input, endtime, timeout)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib64/python3.6/subprocess.py", line 1521, in _communicate
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor with _PopenSelector() as selector:
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor File "/usr/lib64/python3.6/selectors.py", line 348, in __init__
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor self._poll = select.poll()
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor AttributeError: module 'select' has no attribute 'poll'

Quoting Michele:
"""
This is (another) eventlet monkeypatching issue. Reproducer:
import git
import eventlet
from eventlet import timeout as ev_timeout
import sys

eventlet.monkey_patch(
    os=True,
    select=True,
    socket=True,
    thread=False if '--use-debugger' in sys.argv else True,
    time=True)
"""

Version-Release number of selected component (if applicable):
openstack-mistral-executor:20190226.1

How reproducible:
Always

Steps to Reproduce:
1. deploy an overcloud

Actual results:
mistral fails, deployment never proceeds

Expected results:
no failure

[Adriano Petrich (apetrich)]

Did some testing and it works on

eventlet == 0.19.0
But on eventlet 0.20.0 and onwards it breaks

[Adriano Petrich (apetrich)]

We are as we speak on version 0.24.1 the requirements way of setting that eventlet version is not very futureproof

from here: https://github.com/openstack/requirements/blob/master/global-requirements.txt#L58-L59

# NOTE: New versions of eventlet should not be accepted lightly
# as they have earned a reputation of frequently breaking things.
eventlet!=0.18.3,!=0.20.1,!=0.21.0,!=0.23.0 # MIT

[Adriano Petrich (apetrich)]

I have a solution. Somehow subprocess is not being correctly patched by eventlet in the git module.

patching manually on import makes it work like this:

"""
import eventlet
from eventlet import timeout as ev_timeout
from eventlet.green import subprocess
import eventlet.patcher
import sys

eventlet.monkey_patch(
    os=True,
    select=True,
    socket=True,
    thread=False if '--use-debugger' in sys.argv else True,
    time=True)

git = eventlet.import_patched('git', ('subprocess', subprocess))
repo = git.Repo.init("/tmp/foo")

"""

I'm writing a patch for tripleo-common. but I removing the bug from mistral because it is more of an eventlet issue than a mistral one.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/641338

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/641338
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=1af49b1966434411baa1fcb65ffb9953bc91f6a5
Submitter: Zuul
Branch: master

commit 1af49b1966434411baa1fcb65ffb9953bc91f6a5
Author: apetrich <email address hidden>
Date: Wed Mar 6 12:57:42 2019 +0100

    Fix interaction between gitpython and eventlet

    The mistral executor uses eventlet patching, but gitpython's subprocess
    module is not being properly patched. Patching it manually fix the issues

    Change-Id: I49d041b48d9939844bcc3a571819887772caf491
    Closes-bug: #1818757

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.5.0 release.