gate-bandit-integration-* not failing when they should

[Eric Brown]

Bandit has an integration tox job which checks against other projects whether a bandit patch potentially breaks them. I noticed its not working as of late.

When errors occur, they are ignored. Either the exit code is not preserved or something else is happening. The job always shows success.

For example (gate-bandit-integration-keystone):

2016-02-17 19:15:01.574 | + pushd ../keystone
2016-02-17 19:15:01.574 | /home/jenkins/workspace/gate-bandit-integration-keystone/openstack/keystone /home/jenkins/workspace/gate-bandit-integration-keystone/openstack/bandit
2016-02-17 19:15:01.574 | + set +e
2016-02-17 19:15:01.574 | + tox -e bandit --notest
2016-02-17 19:15:01.663 | ERROR: unknown environment 'bandit'
2016-02-17 19:15:01.670 | + .tox/bandit/bin/pip install --force-reinstall -U /home/jenkins/workspace/gate-bandit-integration-keystone/openstack/bandit
2016-02-17 19:15:01.671 | scripts/integration-test.sh: line 48: .tox/bandit/bin/pip: No such file or directory
2016-02-17 19:15:01.671 | + tox -e bandit
2016-02-17 19:15:01.795 | ERROR: unknown environment 'bandit'
2016-02-17 19:15:01.802 | + popd
2016-02-17 19:15:01.802 | /home/jenkins/workspace/gate-bandit-integration-keystone/openstack/bandit
2016-02-17 19:15:01.802 | + [[ 0 -eq 1 ]]
2016-02-17 19:15:01.803 | ___________________________________ summary ____________________________________
2016-02-17 19:15:01.803 | integration: commands succeeded
2016-02-17 19:15:01.803 | congratulations :)

gate-bandit-integration-oslo.vmware:

2016-02-17 19:15:20.354 | >> Issue: [B309:blacklist] Use of HTTPSConnection does not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033
2016-02-17 19:15:20.354 | Severity: Medium Confidence: High
2016-02-17 19:15:20.354 | Location: oslo_vmware/objects/datastore.py:313
2016-02-17 19:15:20.354 | 311 conn = httplib.HTTPConnection(self._server)
2016-02-17 19:15:20.354 | 312 elif self._scheme == 'https':
2016-02-17 19:15:20.354 | 313 conn = httplib.HTTPSConnection(self._server)
2016-02-17 19:15:20.354 | 314 else:
2016-02-17 19:15:20.354 | 315 excep_msg = _("Invalid scheme: %s.") % self._scheme
2016-02-17 19:15:20.354 |
2016-02-17 19:15:20.355 | --------------------------------------------------
2016-02-17 19:15:20.355 | >> Issue: [B112:os_path_join_traversal] Check variables used with os.path.join are trusted or sanitised.
2016-02-17 19:15:20.355 | Severity: Low Confidence: Low
2016-02-17 19:15:20.355 | Location: oslo_vmware/pbm.py:196
2016-02-17 19:15:20.355 | 194 major_minor = '%s.%s' % (major_minor, ver[1])
2016-02-17 19:15:20.355 | 195 curr_dir = os.path.abspath(os.path.dirname(__file__))
2016-02-17 19:15:20.355 | 196 pbm_service_wsdl = os.path.join(curr_dir, 'wsdl', major_minor,
2016-02-17 19:15:20.355 | 197 'pbmService.wsdl')
2016-02-17 19:15:20.355 | 198 if not os.path.exists(pbm_service_wsdl):
2016-02-17 19:15:20.355 | 199 LOG.warning(_LW("PBM WSDL file %s not found."), pbm_service_wsdl)
2016-02-17 19:15:20.356 |
2016-02-17 19:15:20.356 | --------------------------------------------------
2016-02-17 19:15:20.374 | ERROR: InvocationError: '/home/jenkins/workspace/gate-bandit-integration-oslo.vmware/openstack/oslo.vmware/.tox/bandit/bin/bandit -r oslo_vmware -n 5'
2016-02-17 19:15:20.374 | ___________________________________ summary ____________________________________
2016-02-17 19:15:20.374 | ERROR: bandit: commands failed
2016-02-17 19:15:20.382 | + popd
2016-02-17 19:15:20.382 | /home/jenkins/workspace/gate-bandit-integration-oslo.vmware/openstack/bandit
2016-02-17 19:15:20.382 | + [[ 0 -eq 1 ]]
2016-02-17 19:15:20.382 | ___________________________________ summary ____________________________________
2016-02-17 19:15:20.382 | integration: commands succeeded
2016-02-17 19:15:20.382 | congratulations :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/281560

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281560
Committed: https://git.openstack.org/cgit/openstack/bandit/commit/?id=8dd30b82844a464ac7d9a847a400e2af368c1b4e
Submitter: Jenkins
Branch: master

commit 8dd30b82844a464ac7d9a847a400e2af368c1b4e
Author: Eric Brown <email address hidden>
Date: Wed Feb 17 14:41:44 2016 -0800

    Ensure error exit codes fail integrations

    The exit code of sub commands were ignored. As a result all
    integration jobs would pass even when they fail.

    Change-Id: I071283d2737199ed710e246740f68f8e857027f2
    Closes-Bug: #1546772