devstack-gate setup doesn't differentiate git fetch errors from other git errors

[Doug Wiegley]

Using devstack-gate, during the devstack-gate-setup-workspace step, if the git fetch from zuul has an error, the script continues, and you end up unknowingly testing against master instead of the given gerrit change.

[Doug Wiegley (dougwig)]

This isn't quite the same. The dup'ed bug had the tests fail on a git failure. The issue I had is that the test run did NOT fail, and I got a false positive. Note the fatal error below (configuration error, but the same would occur if say, zuul's apache was down). It continued and ran the tests against master, and then reported results as if it had tested the gerrit review:

2014-08-16 01:40:50.405 | + FALLBACK_ZUUL_REF=
2014-08-16 01:40:50.412 | + git_has_branch openstack/neutron master
2014-08-16 01:40:50.414 | + local project=openstack/neutron
2014-08-16 01:40:50.415 | + local branch=master
2014-08-16 01:40:50.416 | + git branch -a
2014-08-16 01:40:50.418 | + grep remotes/origin/master
2014-08-16 01:40:50.419 | + return 0
2014-08-16 01:40:50.420 | + git_fetch_at_ref openstack/neutron refs/zuul/master/Z9a2f87876cc54d7e9fdf7a7c1b27647e
2014-08-16 01:40:50.422 | + local project=openstack/neutron
2014-08-16 01:40:50.423 | + local ref=refs/zuul/master/Z9a2f87876cc54d7e9fdf7a7c1b27647e
2014-08-16 01:40:50.424 | + '[' refs/zuul/master/Z9a2f87876cc54d7e9fdf7a7c1b27647e '!=' '' ']'
2014-08-16 01:40:50.425 | + git fetch http://zuul.openstack.org/p/openstack/neutron refs/zuul/master/Z9a2f87876cc54d7e9fdf7a7c1b27647e
2014-08-16 01:40:50.582 | fatal: repository 'http://zuul.openstack.org/p/openstack/neutron/' not found
2014-08-16 01:40:50.585 | + return 128
2014-08-16 01:40:50.587 | + git_fetch_at_ref openstack/neutron
2014-08-16 01:40:50.588 | + local project=openstack/neutron
2014-08-16 01:40:50.589 | + local ref=
2014-08-16 01:40:50.590 | + '[' '' '!=' '' ']'
2014-08-16 01:40:50.591 | + return 1
2014-08-16 01:40:50.592 | + git_has_branch openstack/neutron master
2014-08-16 01:40:50.597 | + local project=openstack/neutron
2014-08-16 01:40:50.598 | + local branch=master
2014-08-16 01:40:50.599 | + git branch -a
2014-08-16 01:40:50.600 | + grep remotes/origin/master
2014-08-16 01:40:50.602 | + return 0
2014-08-16 01:40:50.605 | + git_checkout openstack/neutron master
2014-08-16 01:40:50.606 | + local project=openstack/neutron
2014-08-16 01:40:50.607 | + local branch=master
2014-08-16 01:40:50.608 | + local reset_branch=master
2014-08-16 01:40:50.610 | + [[ master != \F\E\T\C\H\_\H\E\A\D ]]
2014-08-16 01:40:50.611 | + reset_branch=remotes/origin/master
2014-08-16 01:40:50.612 | + git checkout master
2014-08-16 01:40:50.648 | Already on 'master'
2014-08-16 01:40:50.655 | Your branch is behind 'origin/master' by 282 commits, and can be fast-forwarded.
2014-08-16 01:40:50.656 | (use "git pull" to update your local branch)
2014-08-16 01:40:50.661 | + git reset --hard remotes/origin/master
2014-08-16 01:40:50.733 | HEAD is now at 9bf0e66 Remove auto-generation of db schema from models at startup
2014-08-16 01:40:50.734 | + git clean -x -f -d -q
2014-08-16 01:40:50.745 | + for PROJECT in '$PROJECTS'
2014-08-16 01:40:50.747 | + cd /opt/stack/new
2014-08-16 01:40:50.748 | + '[' -d /opt/git/openstack/marconi ']'
2014-08-16 01:40:50.751 | ++ basename openstack/marconi
2014-08-16 01:40:50.757 | + rsync -a /opt/git/openstack/marconi/ marconi

[Jeremy Stanley (fungi)]

The duped bug was mistaken and the job failed for reasons other than the git errors which were mistakenly assumed to be related (or even indicative of any problem at all).

In this case, it looks like this isn't a log from our (upstream, OpenStack) CI at all, because our jobs should be attempting to retrieve zuul refs from our zuul merger servers not our zuul scheduler server. Where was this observed?

Regardless the design of devstack-gate, as it exists now, assumes that many git fetches will fail and uses those as an indicator that it should fall back to other (usually branch tip) references instead. Yes this does also mean that when you misconfigure the CI this software will not work as expected, but then the same can be said of a lot of software and I'm not convinced that counts as a bug.

[Doug Wiegley (dougwig)]

Hiya. You are correct, it's from a third-party CI, which git pulls devstack-gate.

I'm not claiming that the code must always work in the face of a configuration error. But this particular configuration error highlights what I consider a bug. Namely, that the script doesn't differentiate between intentional errors and real operational errors, resulting in potential false positives.

It's fine if you don't want to fix it, it's your code base, but it's definitely a point of brittleness.

[Jeremy Stanley (fungi)]

Agreed, ideally there could be some way to tell the difference between "Couldn't find remote ref" and other types of git errors (perhaps by checking the return code which is "1" in the normal case and >=128 for things like the above example). However we should avoid things like parsing stderr, I think, which are likely to be brittle in other ways...