branching scripts need to tolerate glitches in opendev gerrit

[Scott Little]

Brief Description
-----------------
During creation of the r/stx.6.0 branch, the opendev gerrit server was failing on a high percentage of all git review and git push commands. The branching scripts need to tolerate glitches by adding delays and retries. Perhaps the high volume of requests over a short time was triggering an anti-spam or anti-DDS function?

Severity
--------
Major

Steps to Reproduce
------------------
create_branches_and_tags.sh --branch=__test__ --remotes=starlingx --manifest --soft-lock-down --default-revision --gitreview-default --lock-down-exclude-remotes=starlingx-docs --manifest

push_branches_tags.sh --branch=__test__ --remotes=starlingx --manifest

Expected Behavior
------------------
branches are pushed to all gits without error

Actual Behavior
----------------
nearly 50% of branches failed to push

Reproducibility
---------------
100%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
Dec 9 2021

Last Pass
---------
N/A

Timestamp/Logs
--------------

Problem running 'git remote update gerrit'
Fetching gerrit
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error: Could not fetch gerrit

..

Problems encountered installing commit-msg hook
The following command failed with exit code 1
    "scp -P29418 <email address hidden>:hooks/commit-msg .git/hooks/commit-msg"
-----------------------
ssh_exchange_identification: read: Connection reset by peer
-----------------------
ERROR: 1: failed to setup git review in /localdisk/designer/slittle1/stx-6.0/cgcs-root/stx/test
slittle1@yow-slittle1-lx2$create_branches_and_tags.sh --branch=r/stx.6.0 --remotes=starlingx --manifest --soft-lock-down --default-revision --gitreview-default --lock-down-exclude-remotes=starlingx-docs

Test Activity
-------------
Build

Workaround
----------
Manual branch creation

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/821475

[Ghada Khalil (gkhalil)]

screening / medium - build/branch tool robustness, so it's sufficient to fix in stx master; will not hold up stx.6.0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/821475
Committed: https://opendev.org/starlingx/root/commit/06537894b2bd9be41dc6d9537d41e0a81b47db7e
Submitter: "Zuul (22348)"
Branch: master

commit 06537894b2bd9be41dc6d9537d41e0a81b47db7e
Author: Scott Little <email address hidden>
Date: Fri Dec 10 16:06:00 2021 -0500

    branching: Add retries to git push/review commands

    Review.opendev.org seem unreliable when hit
    with a lot of transactions in a short time.
    This means that an attempt to create a release branch,
    e.g. r/stx.6.0 failes repeatedly, and requires a lot
    of manual repair.

    By adding retires and delays, we greatly improve
    our odds of success.

    Closes-bug: 1954536
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: Ia03ed58b67c02e6984a7e0ee01c6666027df6cbe