devstack fails to install checked out git repositories with latest version of git

Bug #1968798 reported by Michael Johnson
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
devstack
Expired
Undecided
Unassigned

Bug Description

Currently all devstack jobs using a current version of git (focal is known to have released a version listed below) will fail to install repositories checked out locally due to a security fix[1] in git.

The error reported will look like this (but it is not limited to keystone):

2022-04-12 22:13:00.471769 | controller | Obtaining file:///opt/stack/keystone
2022-04-12 22:13:00.942756 | controller | ERROR: Command errored out with exit status 1:
2022-04-12 22:13:00.942912 | controller | command: /usr/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/opt/stack/keystone/setup.py'"'"'; __file__='"'"'/opt/stack/keystone/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
2022-04-12 22:13:00.943022 | controller | cwd: /opt/stack/keystone/
2022-04-12 22:13:00.943097 | controller | Complete output (16 lines):
2022-04-12 22:13:00.943172 | controller | Error parsing
2022-04-12 22:13:00.943242 | controller | Traceback (most recent call last):
2022-04-12 22:13:00.943312 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/core.py", line 111, in pbr
2022-04-12 22:13:00.943381 | controller | attrs = util.cfg_to_args(path, dist.script_args)
2022-04-12 22:13:00.943493 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/util.py", line 272, in cfg_to_args
2022-04-12 22:13:00.943568 | controller | pbr.hooks.setup_hook(config)
2022-04-12 22:13:00.943637 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/hooks/__init__.py", line 25, in setup_hook
2022-04-12 22:13:00.943703 | controller | metadata_config.run()
2022-04-12 22:13:00.943772 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/hooks/base.py", line 27, in run
2022-04-12 22:13:00.943839 | controller | self.hook()
2022-04-12 22:13:00.943907 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/hooks/metadata.py", line 25, in hook
2022-04-12 22:13:00.943973 | controller | self.config['version'] = packaging.get_version(
2022-04-12 22:13:00.944041 | controller | File "/usr/local/lib/python3.8/dist-packages/pbr/packaging.py", line 872, in get_version
2022-04-12 22:13:00.944108 | controller | raise Exception("Versioning for this project requires either an sdist"
2022-04-12 22:13:00.944188 | controller | Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name keystone was given, but was not able to be found.
2022-04-12 22:13:00.944267 | controller | error in setup command: Error parsing /opt/stack/keystone/setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name keystone was given, but was not able to be found.
2022-04-12 22:13:00.944346 | controller | ----------------------------------------
2022-04-12 22:13:00.944428 | controller | ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The issue is that the repository /opt/stack/keystone is owned by stack:stack but installed under sudo which triggers this new security feature in git to not access the directory.

clarkb was able to reproduce the issue and chown -R /opt/stack/keystone to root allowed the installation.

On Ubuntu focal,
git 1:2.25.1-1ubuntu3.2 will run fine
git 1:2.25.1-1ubuntu3.3 will fail

Installing 3.2 in a test job showed a successful install here: https://review.opendev.org/c/openstack/devstack/+/837634

Related documentation in git can be found here: https://github.com/git/git/blob/master/Documentation/config/safe.txt

[1] https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9

Revision history for this message
Ian Wienand (iwienand) wrote :

https://review.opendev.org/c/openstack/devstack/+/837636

is one potential solution; that chowns the checked-out directories as root:root. it may be a bit whack-a-mole finding places that want to fiddle with the source directories for various reasons.

using "git config" to mark the directories as "safe" is also an option. it doesn't feel like the type of thing we should be doing to a users machine -- but then again devstack takes over everything else anyway ...

Unfortunately, per the docs in the original commit, you can *not* just use "-c" to set one directory as safe from the git command-line. That basically rules out PBR being able to override this (the theory being, if you're telling PBR to install a source tree, then you trust that directory).

Other suggestion has been to install from a shallow clone. Some question over how PBR would cope with this.

The other approach is to completely ditch all this global install and use virtualenvs. Which I think we can all agree is probably a good idea, but has many implications for a) just getting that working in devstack and then b) bringing along every project and its plugins ...

Revision history for this message
Ian Wienand (iwienand) wrote :

I've been unable to make changing ownership work, at this point.

I've proposed the simpler, but uglier, approach of using the "safe" config flag for checkouts in https://review.opendev.org/c/openstack/devstack/+/837659

This seems to work. It will need backporting for grenade, if that's what we want to do.

Revision history for this message
serenico (federico-ressi) wrote :

The reason of the failure is a security check that git does to avoid root user to execute unexpected actions. For example if a malicious guy puts some malicious script in git hooks, when pbr uses git as root to check some thing, because the directory is not owned by root git refuses to do anything instead of riskying to execute any of the hooks scripts. So the new behavior is good thing, bad thing is now using pip as root with pbr running git for stupid things as looking for the git folder or the version of the package is going to install. As a long term solution for DevStack I would install these packages as stack user. But in the while the solution you proposed looks pretty good I think. Cheers

Revision history for this message
Michael Johnson (johnsom) wrote :
Revision history for this message
Martin Kopec (mkopec) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to devstack (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/devstack/+/838722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to devstack (stable/train)

Reviewed: https://review.opendev.org/c/openstack/devstack/+/838722
Committed: https://opendev.org/openstack/devstack/commit/b7f94eafd0f35f3d818deff3fda720387419690a
Submitter: "Zuul (22348)"
Branch: stable/train

commit b7f94eafd0f35f3d818deff3fda720387419690a
Author: Brian Rosmaita <email address hidden>
Date: Wed Apr 20 09:30:43 2022 -0400

    Make neutron-grenade non-voting on stable/train

    The stable/ussuri, stable/train, and stable/stein branches are
    in Extended Maintenance mode and no longer released from, so make
    neutron-grenade non-voting in check and remove it from the
    gate.

    All other grenade jobs are already non-voting in stable/train and
    were removed from the gate four years ago by change I82f539bfa533.

    Change-Id: I21975b270786687149b232866b2009c145ad9915
    Related-bug: #1968798

tags: added: in-stable-train
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

I think we should have workarounds for this in place now, please update if there is still some open issue.

Changed in devstack:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for devstack because there has been no activity for 60 days.]

Changed in devstack:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers