neutron-lib hacking check roll-out breaks consumers

[Boden R]

Since [1] merged, the periodic-neutron-py34-with-neutron-lib-master job is failing with:

2016-08-03 06:12:19.091524 | --- import errors Non-zero exit code (2) from test listing.
2016-08-03 06:12:19.091836 | ---
2016-08-03 06:12:19.093990 | Failed to call load_tests:
2016-08-03 06:12:19.096585 | Traceback (most recent call last):
2016-08-03 06:12:19.110744 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/.tox/py34-neutron-lib-master/lib/python3.4/site-packages/unittest2/loader.py", line 147, in loadTestsFromModule
2016-08-03 06:12:19.114036 | return load_tests(self, tests, pattern)
2016-08-03 06:12:19.124972 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/neutron/tests/unit/hacking/test_checks.py", line 399, in load_tests
2016-08-03 06:12:19.126979 | ignore=('F', 'H104'))
2016-08-03 06:12:19.140594 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/.tox/py34-neutron-lib-master/lib/python3.4/site-packages/flake8/engine.py", line 234, in get_style_guide
2016-08-03 06:12:19.143540 | styleguide = StyleGuide(**kwargs)
2016-08-03 06:12:19.156669 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/.tox/py34-neutron-lib-master/lib/python3.4/site-packages/flake8/engine.py", line 154, in __init__
2016-08-03 06:12:19.162183 | self._styleguide = kwargs.pop('styleguide', NoQAStyleGuide(**kwargs))
2016-08-03 06:12:19.169531 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/.tox/py34-neutron-lib-master/lib/python3.4/site-packages/pep8.py", line 1651, in __init__
2016-08-03 06:12:19.169591 | options.logical_checks = self.get_checks('logical_line')
2016-08-03 06:12:19.169641 | File "/home/jenkins/workspace/periodic-neutron-py34-with-neutron-lib-master/.tox/py34-neutron-lib-master/lib/python3.4/site-packages/pep8.py", line 1747, in get_checks
2016-08-03 06:12:19.169654 | return sorted(checks)
2016-08-03 06:12:19.169673 | TypeError: unorderable types: function() < function()
2016-08-03 06:12:19.169690 | The test run didn't actually run any tests

See [2] for full log

[1] https://review.openstack.org/#/c/333500/
[2] http://logs.openstack.org/periodic/periodic-neutron-py34-with-neutron-lib-master/532d25b/console.html#_2016-08-03_06_12_19_093990

[Boden R (boden)]

I've reproduced this locally and have been digging into root cause.

I believe the issue is related to the fact that the neutron_lib registers entry point checks that are re-registered in neutron. I'll dig deeper and propose a fix.

[Victor Morales (electrocucaracha)]

Hey Boden, I would like to reproduce this one in my local env, can you provide the steps to do it?

[Boden R (boden)]

To repro, need to use neutron master with neutron-lib master + python 3.4.

What I've been doing is:

- Edit neutron/requirements.txt and comment out neutron-lib.
- Run neutron tox for py34:
    tox -e py34
- The venv will setup, but will UTs fail because neutron-lib is not installed.
- Activate py34 venv:
   source ./.tox/py34/bin/activate
- Clone neutron-lib master and install it (with py34 venv active):
   python setup.py install
- Run tools/ostestr_compat_shim.sh
- It will fail as it starts up with the error shown in the bug.

Please note: even if we can get rid of the exception, this behavior in neutron-lib needs to change. Please see the commit message of the patch submitted to fix this bug.

Thanks

[Boden R (boden)]

I'm repurposing this bug for some related work.

In particular we've seen some issues when rolling out new hacking checks in neutron-lib. For example see the discussion in [1]. The work under this bug should look to solidify the hacking check approach taken by neutron-lib including how consumers use it's hacking checks factory in their tox.ini for pep8 gate.

[1] https://review.openstack.org/#/c/360590/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/350723
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=d1a5ea03c28f0da5651c3432072396d1ea6195ba
Submitter: Jenkins
Branch: master

commit d1a5ea03c28f0da5651c3432072396d1ea6195ba
Author: Boden R <email address hidden>
Date: Wed Dec 7 14:11:33 2016 -0700

    Separate hacking check factories

    Neutron-lib adopters should be using neutron-lib's hacking
    checks (factory) to validate compliance, but are susceptible
    to breakage when new checks are released in neutron-lib (see
    partial bug). What we need is a way to roll-out adopter hacking
    checks in a controlled manner, giving consumers the opportunity
    and tools to comply with new checks.

    This patch begins to address this need by:
    - Separating neutron-lib hacking checks into 3 separate factories;
    adopter, incubating and neutron-lib internal (private).
    - Moving any existing checks into their respective factory.
    - Updating the usage docs on how to adopt our hacking checks.
    - Trusty ole unit test; for obvious reasons.

    This patch plays a part in a longer term hacking check strategy:
    - Adopters must run neutron-lib's hacking check factory() or its
    equivalent via their pep8 checks. This is part of their pep8 gate job
    once they update their tox.ini to use neutron's factory().
    - Adopters must run a non-voting neutron-lib-pep8-incubating gate job
    that runs pep8 with neutron-lib's incubating_factory(). This is means
    to warn adopters of upcoming checks without failing their gates.
    This job will be created (by me) assuming reviewers agree to this approach.
    - We'll communicate hacking checks via the standard means (see usage
    docs herein).

    Closure of Bug #1609478 will follow completion of the strategy above.

    Partial-Bug: #1609478
    Change-Id: I5b8d5c41f1bf463109baf9c26ae9d9db5719097d

[Boden R (boden)]

Going to close this one out; it's more or less noise.
The change in [1] addresses the issue, but continue to discuss the best way to deal with hacking checks in lib. For example [2]. That said, I don't see a reason to leave this bug open; it's unnecessary.

[1] https://review.openstack.org/350723/
[2] https://review.openstack.org/421484/