Cloning with GIT_DEPTH setting is broken

[Martin Kopec]

This patch https://review.opendev.org/c/openstack/devstack/+/882299 doesn't work with a GIT_DEPTH=1 setting.

That commit basically changed from

    git clone ... --branch <BRANCH>

to

    git clone ...
    git checkout <BRANCH>

So it's easy for the desired <BRANCH> to be unavailable if GIT_DEPTH is limited.

It was changed (by 882299) because --branch didn't accept hashes anymore (not exactly sure why) but didn't realize that we might be passing --depth argument there.
We'll need to rework that code so that it accepts both, branches and hashes, as well as depth argument.

[Ihar Hrachyshka (ihar-hrachyshka)]

Is there a local.conf that I could use to reproduce this behavior? I would like to take a look but I am not knowledgeable enough about how --depth is used by people, so I'd like to reproduce to understand the problem better. Thanks.

[Martin Kopec (mkopec)]

Hi Ihar,

you don't need a specific local.conf for this. This affects all devstack installations as it's part of the code which downloads all dependencies needed to install an OpenStack environment.

Here's a minimal config file you need: https://docs.openstack.org/devstack/latest/#create-a-local-conf
Or you can use any one from any of the jobs executed in the CI.

Anyway, I played with it without running whole devstack. I just ran 'git clone' with and without --depth, e.g.:
git clone --depth 1 https://opendev.org/openstack/devstack.git devstack

--depth specifies the number of commits you wanna download. By default, git clone, downloads all commit in all branches - devstack has more than 10 thousand commits - if you download just the last commit you'll speed up the cloning process by several seconds, depends also on the speed of the internet connection - in my local env i was able to save 2 seconds (it took 1 second instead of 3). Now imagine, that devstack clones 15 - 20 projects ...

Anyway, the issue in a nutshell was in a case when you downloaded just the latest commit, however, git_ref was set to a specific one - if git_ref != the latest commit => error:

$ git_timed clone $git_clone_flags $git_remote $git_dest
$ cd $git_dest
$ git checkout $git_ref

[Martin Kopec (mkopec)]

Fix proposed:
https://review.opendev.org/c/openstack/devstack/+/889012

[Martin Kopec (mkopec)]

The fix has been merged to master: https://review.opendev.org/c/openstack/devstack/+/889012

We're also backporting this to several stable branches:
https://review.opendev.org/q/I748354964a133e028e12458cc9014d6d014cbdb9

I'm closing this as Fix Released.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890332
Committed: https://opendev.org/openstack/devstack/commit/9b3ac634f54082f1450b33a38abcc8219cd3e96c
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 9b3ac634f54082f1450b33a38abcc8219cd3e96c
Author: jskunda <email address hidden>
Date: Tue Jul 18 09:32:05 2023 +0200

    git: git checkout for a commit hash combinated with depth argument

    This patch: https://review.opendev.org/c/openstack/devstack/+/882299
    provides functionality, that commit hash can be passed as last arugment,
    however when GIT_DEPTH is set, it fails, as in:

    timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
    --depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    not found in upstream origin

    Closes-Bug: #2023020
    Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890333
Committed: https://opendev.org/openstack/devstack/commit/722465ec974a794a455063e42ad7813556e1b93e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 722465ec974a794a455063e42ad7813556e1b93e
Author: jskunda <email address hidden>
Date: Tue Jul 18 09:32:05 2023 +0200

    git: git checkout for a commit hash combinated with depth argument

    This patch: https://review.opendev.org/c/openstack/devstack/+/882299
    provides functionality, that commit hash can be passed as last arugment,
    however when GIT_DEPTH is set, it fails, as in:

    timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
    --depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    not found in upstream origin

    Closes-Bug: #2023020
    Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/888752
Committed: https://opendev.org/openstack/devstack/commit/0a1ea3ed531a0c331e73d57b318b1d7450a8848f
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 0a1ea3ed531a0c331e73d57b318b1d7450a8848f
Author: jskunda <email address hidden>
Date: Tue Jul 18 09:32:05 2023 +0200

    git: git checkout for a commit hash combinated with depth argument

    This patch: https://review.opendev.org/c/openstack/devstack/+/882299
    provides functionality, that commit hash can be passed as last arugment,
    however when GIT_DEPTH is set, it fails, as in:

    timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
    --depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    not found in upstream origin

    Closes-Bug: #2023020
    Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/890334
Committed: https://opendev.org/openstack/devstack/commit/431b5c2ed5aec2dd31effe1b22ad7fb8eee0aca4
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 431b5c2ed5aec2dd31effe1b22ad7fb8eee0aca4
Author: jskunda <email address hidden>
Date: Tue Jul 18 09:32:05 2023 +0200

    git: git checkout for a commit hash combinated with depth argument

    This patch: https://review.opendev.org/c/openstack/devstack/+/882299
    provides functionality, that commit hash can be passed as last arugment,
    however when GIT_DEPTH is set, it fails, as in:

    timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
    --depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
    not found in upstream origin

    Closes-Bug: #2023020
    Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/devstack/+/890338
Reason: stable/victoria branch of openstack/devstack is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/890337
Reason: stable/wallaby branch of openstack/devstack is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.