Problem with tripleo_common git functionalities and eventlet 0.25.1

[Alfredo Moralejo]

When trying to update eventlet to 0.25.1 in RDO Train, as defined in upper-constraints.txt (and needed for other projects), tripleo is failing with following error:

http://logs.rdoproject.org/09/22509/1/check/rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset010/19edef6/logs/undercloud/var/log/containers/mistral/executor.log.txt.gz

2019-09-24 06:28:15.133 8 INFO tripleo_common.utils.config.Config [req-c5b520a9-783b-4877-b70c-2d3c8a729cf6 8e4d825f7721459c86d2312cb9b6fcb7 5c3f07657461453288a6c6b9e82b5f30 - default default] Snapshotting /tmp/tripleo-cPjiQX-config
2019-09-24 06:28:15.175 8 WARNING mistral.executors.default_executor [req-c5b520a9-783b-4877-b70c-2d3c8a729cf6 8e4d825f7721459c86d2312cb9b6fcb7 5c3f07657461453288a6c6b9e82b5f30 - default default] The action raised an exception [action_ex_id=1a5f677e-b89d-4db3-ab26-516a0706c365, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{u'container_config': u'overcloud-config', u'container': u'overcloud'}']
 'Head' object has no attribute 'startswith': AttributeError: 'Head' object has no attribute 'startswith'
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/actions/config.py", line 87, in run
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor commit_message=message)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/utils/config.py", line 543, in download_config
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor self.snapshot_config_dir(git_repo, commit_message)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/utils/config.py", line 162, in snapshot_config_dir
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor commit = repo.index.commit(commit_message)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/git/index/base.py", line 924, in commit
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor head, author=author, committer=committer)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/git/objects/commit.py", line 375, in create_from_tree
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor master = git.refs.Head.create(repo, repo.head.ref, new_commit, logmsg="commit (initial): %s" % message)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 527, in create
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor return cls._create(repo, path, cls._resolve_ref_on_create, reference, force, logmsg)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 473, in _create
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor full_ref_path = cls.to_full_path(path)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/git/refs/symbolic.py", line 402, in to_full_path
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor if not path.startswith(cls._common_path_default + "/"):
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/site-packages/gitdb/util.py", line 239, in __getattr__
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor return object.__getattribute__(self, attr)
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor AttributeError: 'Head' object has no attribute 'startswith'
2019-09-24 06:28:15.175 8 ERROR mistral.executors.default_executor

I can easily reproduce in a virtualenv where i've installed tripleo_common with:

rm -rf /tmp/testrepo
mkdir /tmp/testrepo
touch /tmp/testrepo/file1

from tripleo_common.utils.safe_import import git
repo = git.Repo.init('/tmp/testrepo')
repo.git.add('.')
commit = repo.index.commit('test message')

with eventlet 0.25.1 installed i get same error:

>>> commit = repo.index.commit('test message')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/test1/lib/python2.7/site-packages/git/index/base.py", line 950, in commit
    author_date=author_date, commit_date=commit_date)
  File "/tmp/test1/lib/python2.7/site-packages/git/objects/commit.py", line 397, in create_from_tree
    master = git.refs.Head.create(repo, repo.head.ref, new_commit, logmsg="commit (initial): %s" % message)
  File "/tmp/test1/lib/python2.7/site-packages/git/refs/symbolic.py", line 546, in create
    return cls._create(repo, path, cls._resolve_ref_on_create, reference, force, logmsg)
  File "/tmp/test1/lib/python2.7/site-packages/git/refs/symbolic.py", line 491, in _create
    full_ref_path = cls.to_full_path(path)
  File "/tmp/test1/lib/python2.7/site-packages/git/refs/symbolic.py", line 423, in to_full_path
    if not path.startswith(cls._common_path_default + "/"):
  File "/tmp/test1/lib/python2.7/site-packages/gitdb/util.py", line 255, in __getattr__
    return object.__getattribute__(self, attr)
AttributeError: 'Head' object has no attribute 'startswith'
>>>

While it works with eventlet 0.24.1.

Apparently that safe_load was introduced because of some issue with eventlet and git https://opendev.org/openstack/tripleo-common/commit/1af49b1966434411baa1fcb65ffb9953bc91f6a5.

I'm not sure if, with the new version of eventlet it can be avoided, and also we need to make it work with 0.24.1 and 0.25.1 for transition.

[Adriano Petrich (apetrich)]

It is related to version 0.25 of the eventlet

$ pip install eventlet==0.20
$ ./reset.sh && python test.py

$ pip install eventlet==0.21
$ ./reset.sh && python test.py

$ pip install eventlet==0.22
$ ./reset.sh && python test.py

$ pip install eventlet==0.23
$ ./reset.sh && python test.py

$ pip install eventlet==0.24
$ ./reset.sh && python test.py

$ pip install eventlet==0.25
$ ./reset.sh && python test.py
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    commit = repo.index.commit('test message')
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/git/index/base.py", line 950, in commit
    author_date=author_date, commit_date=commit_date)
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/git/objects/commit.py", line 397, in create_from_tree
    master = git.refs.Head.create(repo, repo.head.ref, new_commit, logmsg="commit (initial): %s" % message)
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/git/refs/symbolic.py", line 546, in create
    return cls._create(repo, path, cls._resolve_ref_on_create, reference, force, logmsg)
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/git/refs/symbolic.py", line 491, in _create
    full_ref_path = cls.to_full_path(path)
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/git/refs/symbolic.py", line 423, in to_full_path
    if not path.startswith(cls._common_path_default + "/"):
  File "/home/frac/Code/test_git_behaviour/foobar/lib/python2.7/site-packages/gitdb/util.py", line 255, in __getattr__
    return object.__getattribute__(self, attr)
AttributeError: 'Head' object has no attribute 'startswith'
(foobar) [frac@snisni test_git_behaviour]$

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684282

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adriano Petrich (<email address hidden>) on branch: master
Review: https://review.opendev.org/684282
Reason: Version 0.25.1 is required for another project so a workaround has to be devised

[Adriano Petrich (apetrich)]

The version 0.25.1 is required for other projects, looking for an alternative

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684380

[Tim Burke (1-tim-z)]

FWIW, I'm fairly certain https://github.com/eventlet/eventlet/commit/05d613d8ed123b87f64f8d1987968c5bf825b07a is the change to eventlet's patcher that broke this -- but I'm still not sure how best to fix it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684380
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=26672e83ab1bd1a932d275dfd244fe20749e3b1e
Submitter: Zuul
Branch: master

commit 26672e83ab1bd1a932d275dfd244fe20749e3b1e
Author: apetrich <email address hidden>
Date: Tue Sep 24 13:20:46 2019 +0200

    Make gitpython and eventlet work with eventlet 0.25.1

    Version 0.25 is having a bad interaction with python git.
    that is due to the way that eventlet unloads some modules now.
    Changed to use the inject method that supports what we need intead
    of the imported_patched that was having the problem

    Change-Id: I79894d4f711c64f536593fffcb6959df97c38838
    Closes-bug: #1845181

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 11.3.0 release.