python33 jobs are breaking

[Davanum Srinivas (DIMS)]

Example : http://logs.openstack.org/87/71187/3/check/gate-python-cinderclient-python33/a982b60/console.html

2014-02-07 01:22:23.598 | Cleaning up...
2014-02-07 01:22:23.598 | Exception:
2014-02-07 01:22:23.598 | Traceback (most recent call last):
2014-02-07 01:22:23.598 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/basecommand.py", line 134, in main
2014-02-07 01:22:23.598 | status = self.run(options, args)
2014-02-07 01:22:23.599 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/commands/install.py", line 236, in run
2014-02-07 01:22:23.599 | requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-02-07 01:22:23.599 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/req.py", line 1139, in prepare_files
2014-02-07 01:22:23.599 | req_to_install.assert_source_matches_version()
2014-02-07 01:22:23.599 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/req.py", line 394, in assert_source_matches_version
2014-02-07 01:22:23.599 | version = self.installed_version
2014-02-07 01:22:23.599 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/req.py", line 390, in installed_version
2014-02-07 01:22:23.600 | return self.pkg_info()['version']
2014-02-07 01:22:23.600 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/req.py", line 357, in pkg_info
2014-02-07 01:22:23.600 | data = self.egg_info_data('PKG-INFO')
2014-02-07 01:22:23.600 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages/pip/req.py", line 297, in egg_info_data
2014-02-07 01:22:23.600 | data = fp.read()
2014-02-07 01:22:23.600 | File "/home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/encodings/ascii.py", line 26, in decode
2014-02-07 01:22:23.601 | return codecs.ascii_decode(input, self.errors)[0]
2014-02-07 01:22:23.601 | UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9055: ordinal not in range(128)

Log stash query:
"UnicodeDecodeError: 'ascii' codec can't decode byte"

[Davanum Srinivas (DIMS) (dims-v)]

I suspect this is coming from latest pbr 0.6 uploaded on 2014-02-03

Downloading/unpacking pip>=1.0 from <a href='http://pypi.openstack.org/openstack/pip/pip-1.5.2.tar.gz'>http://pypi.openstack.org/openstack/pip/pip-1.5.2.tar.gz</a> (from pbr>=0.5.21,&lt;1.0->-r /home/jenkins/workspace/gate-python-cinderclient-python33/requirements.txt (line 1))

[Davanum Srinivas (DIMS) (dims-v)]

looks like we bumped up pbr yesterday
https://github.com/openstack/requirements/commit/ea4a0e6040a08354f5b527e2a434f50d5ddf36de

[Jeremy Stanley (fungi)]

I don't think the pbr minimum version change is involved... we didn't change the upper limit and the new release was 10 days ago.

Instead, this seems to be breaking in setuptools in a Python 3000 specific fashion (byte decoding assumptions on defaulting to ASCII), and setuptools just released 2.1.2 in the past 24 hours.

[Jeremy Stanley (fungi)]

A bit of extra detail...

We're running pip 1.4.1 in the virtualenv where this error occurs:

fungi@precisepy3k-1:~$ /home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/bin/pip --version
pip 1.4.1 from /home/jenkins/workspace/gate-python-cinderclient-python33/.tox/py33/lib/python3.3/site-packages (python 3.3)

And the setuptools 2.0.2 PKG-INFO file was indeed ASCII while 2.1.2's is not:

fungi@static:~$ file setuptools-2.[01].2/setuptools.egg-info/PKG-INFO
setuptools-2.0.2/setuptools.egg-info/PKG-INFO: ASCII English text, with CRLF line terminators
setuptools-2.1.2/setuptools.egg-info/PKG-INFO: UTF-8 Unicode English text, with CRLF line terminators

It looks like the addition of Tarek Ziadé to the credits is triggering this.

[Jeremy Stanley (fungi)]

Actually, this could very well be pbr's fault after all, but I don't think it's related to the pbr 0.6 release specifically. In a bare virtualenv with python 3.3 and pip 1.4.1 I can upgrade setuptools to 2.1.2 with no problem.

Though I also can't seem to recreate this installing the requirements and test-requirements for python-cinderclient into a py33 venv either...

[Jeremy Stanley (fungi)]

Okay, I've confirmed that on an Ubuntu 12.04 LTS system with Python 3.3.2 and empty pip cache I can recreate the problem in a clean clone of python-cinderclient with tox -epy33 (and not doing things to force use of the OpenStack PyPI mirror either).

[Jeremy Stanley (fungi)]

The following trivial change allows tox -epy33 to succeed:

diff --git a/tox.ini b/tox.ini
index fcd7d91..3bab879 100644
--- a/tox.ini
+++ b/tox.ini
@@ -10,7 +10,6 @@ install_command = pip install -U {opts} {packages}
 setenv = VIRTUAL_ENV={envdir}
          LANG=en_US.UTF-8
          LANGUAGE=en_US:en
- LC_ALL=C

 deps = -r{toxinidir}/requirements.txt
        -r{toxinidir}/test-requirements.txt

[Jeremy Stanley (fungi)]

Distilled to a simpler example:

----

$ virtualenv -p python3.3 foo
Already using interpreter /usr/bin/python3.3
Using base prefix '/usr'
New python executable in foo/bin/python3.3
Also creating executable in foo/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
$ env LC_ALL=C foo/bin/pip install -U setuptools
Downloading/unpacking setuptools from https://pypi.python.org/packages/source/s/
setuptools/setuptools-2.1.2.tar.gz#md5=3fb7b3abb02d1d1eb9dc45e1c53e1539
  Downloading setuptools-2.1.2.tar.gz (785kB): 785kB downloaded
  Running setup.py egg_info for package setuptools

Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/req.py", line 1139, in prepare_files
    req_to_install.assert_source_matches_version()
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/req.py", line 394, in assert_source_matches_version
    version = self.installed_version
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/req.py", line 390, in installed_version
    return self.pkg_info()['version']
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/req.py", line 357, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/home/fungi/foo/lib/python3.3/site-packages/pip/req.py", line 297, in egg_info_data
    data = fp.read()
  File "/home/fungi/foo/lib/python3.3/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9055: ordinal not in range(128)

Storing complete log in /home/fungi/.pip/pip.log

----

Note that this happens with pip 1.4.1 even using latest virtualenv 1.11.2, but upgrading pip to a minimum of 1.5 seems to fix the problem.

[Noorul Islam K M (noorul)]

@fungi

In your previous comment, I see that you created a virtualenv, but I don't see you activating it before running the test.

[Jeremy Stanley (fungi)]

Running any python executable from the bin directory of a virtualenv is equivalent to a one-time activation of that virtualenv effective only for that child process. So the foo/bin/pip command above is equivalent to:

    source foo/bin/activate
    env LC_ALL=C pip install -U setuptools
    deactivate

[Monty Taylor (mordred)]

I am in favor of removing the LC_ALL=C lines. In fact, I'm also in favor of removing the LANG lines. We have no documentation on why we think we need them.

[Jeremy Stanley (fungi)]

Let the hunt begin. Setting to invalid in infra and adding bugtasks for all the other projects affected.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72185
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=f6fdff657f079c83b2c3702380218c2b546ee632
Submitter: Jenkins
Branch: master

commit f6fdff657f079c83b2c3702380218c2b546ee632
Author: Russell Bryant <email address hidden>
Date: Sun Feb 9 17:49:29 2014 -0500

    Fix python 3.3 unit test job

    Change-Id: Ibb253e51ec2412d71d04a1994cef4fb60e77f234
    Closes-bug: 1277495

[Jeremy Stanley (fungi)]

In retrospect, I'm not adding all the OpenStack projects which are affected into the bug metadata. LP will break before I get a fraction of the way through the 50 I found. Since Russell looks like he's going to run with the novaclient fix, I'll just propose the other 49:

openstack/api-site
openstack/ceilometer
openstack/cinder
openstack/compute-api
openstack/database-api
openstack-dev/hacking
openstack-dev/pbr
openstack/diskimage-builder
openstack/gantt
openstack/glance
openstack/governance
openstack/heat-cfntools
openstack/identity-api
openstack/image-api
openstack-infra/elastic-recheck
openstack-infra/gear
openstack-infra/git-review
openstack-infra/os-loganalyze
openstack-infra/release-tools
openstack-infra/reviewstats
openstack-infra/storyboard-webclient
openstack/ironic
openstack/keystone
openstack/netconn-api
openstack/nova
openstack/object-api
openstack/openstack-doc-tools
openstack/openstack-manuals
openstack/operations-guide
openstack/os-apply-config
openstack/os-collect-config
openstack/oslo.version
openstack/os-refresh-config
openstack/python-ceilometerclient
openstack/python-cinderclient
openstack/python-glanceclient
openstack/python-ironicclient
openstack/python-keystoneclient
openstack/python-neutronclient
openstack/python-openstackclient
openstack/python-swiftclient
openstack/python-tuskarclient
openstack/requirements
openstack/swift-bench
openstack/taskflow
openstack/tempest
openstack/tripleo-incubator
openstack/tuskar
openstack/volume-api

I don't think we need to worry about backporting to previous releases since this is really only cropping up with Python 3.3 tests anyway.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72215

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72216

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72217

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72218

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72235

[Jeremy Stanley (fungi)]

Also openstack-dev/cookiecutter, but just inside its new project template.

[Noorul Islam K M (noorul)]

solum https://review.openstack.org/#/c/72281/

python-solumclient https://review.openstack.org/#/c/72283/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72215
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=799758410f36751dedf71c52d01a86c3765b9666
Submitter: Jenkins
Branch: master

commit 799758410f36751dedf71c52d01a86c3765b9666
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 03:01:36 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: If453210cb0f9117a445a464e18b842a82cace866
    Closes-Bug: #1277495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72235
Committed: https://git.openstack.org/cgit/openstack/python-cinderclient/commit/?id=7914e46b03614dde9ae4604cfe35b283d3a55253
Submitter: Jenkins
Branch: master

commit 7914e46b03614dde9ae4604cfe35b283d3a55253
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 03:09:27 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: Ibd505447de74f13d90b9ff77dfd8c6b30c4c12cb
    Closes-Bug: #1277495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72217
Committed: https://git.openstack.org/cgit/openstack-infra/git-review/commit/?id=8fd1a948c31e2e94f636d75644e34da7a5406ffb
Submitter: Jenkins
Branch: master

commit 8fd1a948c31e2e94f636d75644e34da7a5406ffb
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 03:02:11 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: I8747a52141321549c56214710b70c5838aacc84e
    Closes-Bug: #1277495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72207
Committed: https://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=ca67985ff37ce602d1892d958d0e7d48b2a6984a
Submitter: Jenkins
Branch: master

commit ca67985ff37ce602d1892d958d0e7d48b2a6984a
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 02:59:04 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: Ic09e1f7f90216064d550bc06c6301d5e1779b5aa
    Closes-Bug: #1277495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72216
Committed: https://git.openstack.org/cgit/openstack-infra/gear/commit/?id=5a2c469895c87bc465fde00445e15d8639f93db1
Submitter: Jenkins
Branch: master

commit 5a2c469895c87bc465fde00445e15d8639f93db1
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 03:01:56 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: I8666ddf1aa3f4f0a71b108efe66ab36d8033813c
    Closes-Bug: #1277495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72218
Committed: https://git.openstack.org/cgit/openstack-infra/os-loganalyze/commit/?id=a57da743b8b3113bcb4d5045e57924a8d2ef6d64
Submitter: Jenkins
Branch: master

commit a57da743b8b3113bcb4d5045e57924a8d2ef6d64
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 10 03:02:34 2014 +0000

    Remove tox locale overrides

    * tox.ini: The LANG, LANGUAGE and LC_ALL environment overrides were
    introduced originally during the testr migration in an attempt to be
    conservative about the possibility that locale settings in the
    calling environment could cause consistency problems for test runs.
    In actuality, this should be unnecessary and any place where it does
    cause issues ought to be considered an actual bug. Also, having
    these in the configuration actively causes older pip to have
    problems with non-ASCII content in some package metadata files under
    Python 3, so drop it now.

    Change-Id: I583ec3eb9ca50750b7b76c90822f1af7f3eb07a0
    Closes-Bug: #1277495

[melanie witt (melwitt)]

novaclient 2.16.0 released on 2/26/2014

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.