gate-keystone-pep8 fail with InvocationError

[David Jia]

I have seen this after Jenkins verify my patchs:https://review.openstack.org/#/c/31074/ due to code "from keystone.openstack.common.gettextutils import _", but this is right code, it should pass pep8 checking

 using tox.ini: /home/jenkins/workspace/gate-keystone-pep8/tox.ini
2013-05-31 08:27:24.583 | using tox-1.4.3 from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc
2013-05-31 08:27:24.583 | GLOB sdist-make: /home/jenkins/workspace/gate-keystone-pep8/setup.py
2013-05-31 08:27:24.593 | /home/jenkins/workspace/gate-keystone-pep8$ /usr/bin/python /home/jenkins/workspace/gate-keystone-pep8/setup.py sdist --formats=zip --dist-dir /home/jenkins/workspace/gate-keystone-pep8/.tox/dist >/home/jenkins/workspace/gate-keystone-pep8/.tox/log/tox-0.log
2013-05-31 08:27:27.169 | pep8 create: /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8
2013-05-31 08:27:27.179 | /home/jenkins/workspace/gate-keystone-pep8/.tox$ /usr/bin/python2.7 /usr/local/lib/python2.7/dist-packages/virtualenv.py --distribute pep8 >/home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/log/pep8-0.log
2013-05-31 08:27:29.035 | pep8 installdeps: -r/home/jenkins/workspace/gate-keystone-pep8/requirements.txt, -r/home/jenkins/workspace/gate-keystone-pep8/test-requirements.txt
2013-05-31 08:27:29.042 | /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/log$ /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/bin/pip install -r/home/jenkins/workspace/gate-keystone-pep8/requirements.txt -r/home/jenkins/workspace/gate-keystone-pep8/test-requirements.txt >/home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/log/pep8-1.log
2013-05-31 08:28:28.338 | pep8 inst: /home/jenkins/workspace/gate-keystone-pep8/.tox/dist/keystone-2013.2.b1.30.g0b936d4.zip
2013-05-31 08:28:28.363 | /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/log$ /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/bin/pip install /home/jenkins/workspace/gate-keystone-pep8/.tox/dist/keystone-2013.2.b1.30.g0b936d4.zip >/home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/log/pep8-2.log
2013-05-31 08:28:30.454 | pep8 runtests: commands[0]
2013-05-31 08:28:30.462 | /home/jenkins/workspace/gate-keystone-pep8$ /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/bin/flake8
2013-05-31 08:28:40.991 | ./keystone/exception.py:20:1: H302 import only modules.'from keystone.openstack.common.gettextutils import _' does not import a module
2013-05-31 08:28:40.991 | from keystone.openstack.common.gettextutils import _
2013-05-31 08:28:40.991 | ^
2013-05-31 08:28:41.202 | ERROR: InvocationError: '/home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/bin/flake8'
2013-05-31 08:28:41.202 | pep8 runtests: commands[1]
2013-05-31 08:28:41.208 | /home/jenkins/workspace/gate-keystone-pep8$ /home/jenkins/workspace/gate-keystone-pep8/.tox/pep8/bin/flake8 --filename=keystone* bin
2013-05-31 08:28:41.774 | ___________________________________ summary ____________________________________
2013-05-31 08:28:41.774 | ERROR: pep8: commands failed
2013-05-31 08:28:41.814 | Build step 'Execute shell' marked build as failure
2013-05-31 08:28:42.202 | [SCP] Connecting to static.openstack.org
2013-05-31 08:28:42.830 | [SCP] Trying to create /srv/static/logs/31074/2/check/gate-keystone-pep8
2013-05-31 08:28:42.837 | [SCP] Trying to create /srv/static/logs/31074/2/check/gate-keystone-pep8/5415
2013-05-31 08:28:42.858 | [SCP] Copying console log.
2013-05-31 08:28:43.062 | Notifying upstream projects of job completion
2013-05-31 08:28:43.062 | Finished: FAILURE

[Dolph Mathews (dolph)]

See the exception:

  H302 import only modules.'from keystone.openstack.common.gettextutils import _' does not import a module

'_' is not a module

[David Jia (jiataotj)]

Hi Dolph,

    I'd like reopen this bug, because it occurs again and "from keystone.openstack.common.gettextutils import _" is a legal code line, please refer nova/openstack/common/log.py(https://github.com/openstack/nova/blob/master/nova/openstack/common/log.py), there is a same code line in it, it's "from nova.openstack.common.gettextutils import _"

[Dolph Mathews (dolph)]

This is definitely not a bug in keystone - moved to hacking for discussion.

[David Jia (jiataotj)]

"from keystone.openstack.common.gettextutils import _" is used in keystone/openstack/common/policy.py too, except that other same code line such as:"from nova.openstack.common.gettextutils import _", "from ceilometer.openstack.common.gettextutils import _" and so on, exist in other components's code a lot. They are legal code, it shouldn't be recognized as an error by pep8 and flake8.

[Chmouel Boudjnah (chmouel)]

there isn't some magic incantation to give to tox.ini to ignore such things ? like :

[flake8]
builtins = _

[David Jia (jiataotj)]

I checked keystone/tox.ini,the configuration "builtins = _" exist

The most wired thing is, when I do keystone pep8 checking, there isn't any error reported against "from keystone.openstack.common.gettextutils import _", but Jenkins pep8 checking filter the code and throw the error. It's strange

[Brant Knudson (blk-u)]

I think that the tests in hacking/core.py aren't honoring # noqa, they need to use noqa(line).

[Brant Knudson (blk-u)]

Proposed fix: https://review.openstack.org/#/c/34019/

[Joe Gordon (jogo)]

I am not sure why projects besides are using "from PROJECT.openstack.common.gettextutils import _" as nova never uses it, instead it uses 'nova/cmd/__init__.py:gettextutils.install('nova')'

That being said this still seems like a valid bug

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34019
Committed: http://github.com/openstack-dev/hacking/commit/d728031755269d352ea43c1ea4fb59fd0eceebd5
Submitter: Jenkins
Branch: master

commit d728031755269d352ea43c1ea4fb59fd0eceebd5
Author: Brant Knudson <email address hidden>
Date: Fri Jun 21 12:20:50 2013 -0500

    Ignore import lines with noqa

    flake8 says that lines with '# noqa' should be ignored, but the
    import rules checker isn't ignoring those lines.

    This change makes it so that import lines with # noqa are ignored.

    Change-Id: Icb2c3eaf0cec87e18bc3c6dcf28f8e2642ec23a2
    Fixes: bug 1186188

[David Jia (jiataotj)]

The Jenkins shouldn't raise error against code line "from keystone. openstack.common.gettextutils import _", the issue still exist and blocks code review: https://review.openstack.org/#/c/31074/

[Joe Gordon (jogo)]

David, The issue still exists, because we have not released the new version of hacking with the fix yet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: 0.5.x
Review: https://review.openstack.org/34615

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34615
Committed: http://github.com/openstack-dev/hacking/commit/8f349ad8cf7db0d1c74005173b7713b301671d9c
Submitter: Jenkins
Branch: 0.5.x

commit 8f349ad8cf7db0d1c74005173b7713b301671d9c
Author: Brant Knudson <email address hidden>
Date: Fri Jun 21 12:20:50 2013 -0500

    Ignore import lines with noqa

    flake8 says that lines with '# noqa' should be ignored, but the
    import rules checker isn't ignoring those lines.

    This change makes it so that import lines with # noqa are ignored.

    Change-Id: Icb2c3eaf0cec87e18bc3c6dcf28f8e2642ec23a2
    Fixes: bug 1186188
    (cherry picked from commit d728031755269d352ea43c1ea4fb59fd0eceebd5)

[David Jia (jiataotj)]

vesrion 0.5.6 still raise error against "from keystone.openstack.common.gettextutils import _", I'll add "# flake8: noqa" as a work around into the module to avoid the bug