[tox] ImportError: No module named config

[Florent Flament]

When trying and running tox on Nova, tests are failing because of missing oslo.config package.

Although oslo.config is specified in the requirements.txt file and pip seems to be successfull in installing oslo.config, the directory where it should be isn't created.

I'm using Nova trunk version 23a0bea232b42df3c26425d7164c779c82fa7a77

Here's my trace:

(cleanenv)ubuntu@flstack2:/opt/stack/nova$ pip freeze
argparse==1.2.1
wsgiref==0.1.2
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ git pull
Already up-to-date.
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ aptitude search oslo
p srvadmin-oslog - Dell OpenManage Server Administrator OS Logging Control
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ rm -r .tox/py27/
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ tox -e py27
py27 create: /opt/stack/nova/.tox/py27
py27 installdeps: -r/opt/stack/nova/requirements.txt, -r/opt/stack/nova/test-requirements.txt
py27 develop-inst: /opt/stack/nova
py27 runtests: commands[0] | python -m nova.openstack.common.lockutils python setup.py test --slowest --testr-args=
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 29, in <module>
    from oslo.config import cfg
ImportError: No module named config
ERROR: InvocationError: '/opt/stack/nova/.tox/py27/bin/python -m nova.openstack.common.lockutils python setup.py test --slowest --testr-args='
__________________________________________________________ summary ___________________________________________________________
ERROR: py27: commands failed
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ ls -d .tox/py27/lib/python2.7/site-packages/oslo*
.tox/py27/lib/python2.7/site-packages/oslo
.tox/py27/lib/python2.7/site-packages/oslo.rootwrap-1.0.0-py2.7.egg-info
.tox/py27/lib/python2.7/site-packages/oslo.rootwrap-1.0.0-py2.7-nspkg.pth
.tox/py27/lib/python2.7/site-packages/oslo.sphinx-1.1-py2.7.egg-info
.tox/py27/lib/python2.7/site-packages/oslo.sphinx-1.1-py2.7-nspkg.pth
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ ls .tox/py27/lib/python2.7/site-packages/oslo
rootwrap sphinx
(cleanenv)ubuntu@flstack2:/opt/stack/nova$ source .tox/py27/bin/activate
(py27)ubuntu@flstack2:/opt/stack/nova$ pip freeze | grep oslo.config
-e git+https://github.com/openstack/oslo.config.git@03930e31965524cb90279b5d9e793c6825791b54#egg=oslo.config-master
(py27)ubuntu@flstack2:/opt/stack/nova$ grep oslo.config requirements.txt
oslo.config>=1.2.0
(py27)ubuntu@flstack2:/opt/stack/nova$

[Florent Flament (florentflament)]

The issue is due to Nova's tox.ini using site packages (sitepackages=True)
https://github.com/openstack/nova/blob/master/tox.ini#L7

With this option, we loose some of the benefits of virtualenv, consisting in isolating packages between the environment where tests are launched and site's packages.

For instance, other Openstack projects tox.ini don't use site packages.
https://github.com/openstack/keystone/blob/master/tox.ini#L6
https://github.com/openstack/cinder/blob/master/tox.ini#L6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68056

[ZhiQiang Fan (aji-zqfan)]

well, neutron does too: https://github.com/openstack/neutron/blob/master/tox.ini#L17

This may need some discussion in mailing list

[Florent Flament (florentflament)]

Thanks to point that out ZhiQiang,

However, in Neutron the `sitepackages = True` option is not in the [testenv] section (it is in the [tox:jenkins] one), so it doesn't make tests fail, when launched locally.

[Florent Flament (florentflament)]

As pointed about by Joe Gordon ( https://review.openstack.org/68056 ), `site-packages = True` is set to allow using the system wide libvirt python library (provided by package python-libvirt - installable through apt-get but not available with pip).

There is an alternative pip package named libvirt-python, that I couldn't have running yet.

[Florent Flament (florentflament)]

An issue arise when trying to use system-packages. If a package (for instance oslo) is available at several different locations, with different modules inside, then some of the modules may not be available to be imported.

>>> pprint(sys.path)
['',
...
 '/opt/stack/nova/.tox/py27/local/lib/python2.7/site-packages',
 '/opt/stack/nova/.tox/py27/lib/python2.7/site-packages',
 '/opt/stack/oslo.config',
 '/opt/stack/oslo.messaging',
...
 '/usr/lib/pymodules/python2.7']
>>> import oslo
>>> pprint(oslo.__path__)
['/opt/stack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo',
 '/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo']
>>> import oslo.config
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named config
>>> oslo.__path__.append('/opt/stack/oslo.config/oslo')
>>> pprint(oslo.__path__)
['/opt/stack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo',
 '/opt/stack/nova/.tox/py27/lib/python2.7/site-packages/oslo',
 '/opt/stack/oslo.config/oslo']
>>> import oslo.config
>>>

[Florent Flament (florentflament)]

libvirt-python (when installing from pip) requires package libvirt-dev to be installed.

Still, we are stuck with current versions of libvirt-python available on pypi:

* version 1.2.0 and 1.2.1 of libvirt-python are available through pip

(debug)ubuntu@flstack2:~$ pip install libvirt-python==0
Downloading/unpacking libvirt-python==0
  Could not find a version that satisfies the requirement libvirt-python==0 (from versions: 1.2.0, 1.2.1)
Cleaning up...
No distributions matching the version for libvirt-python==0
Storing complete log in /home/ubuntu/.pip/pip.log

* libvirt-python-1.2.0 requires at least libvirt-dev version 0.9.11

---- extract from libvirt-python-1.2.0/setup.py ----
MIN_LIBVIRT = "0.9.11"
MIN_LIBVIRT_LXC = "1.0.2"

* libvirt-dev (and libvirt) package available on ubuntu precise is 0.9.8

(debug)ubuntu@flstack2:~$ apt-cache policy libvirt-dev
libvirt-dev:
  Installed: 0.9.8-2ubuntu17.16
  Candidate: 0.9.8-2ubuntu17.16
  Version table:
 *** 0.9.8-2ubuntu17.16 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.8-2ubuntu17.13 0
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
     0.9.8-2ubuntu17 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

[Florent Flament (florentflament)]

Seen on IRC with danbp (OFTC - #virt), libvirt-python can only work with libvirt >= 0.9.11 .

[Kai Qiang Wu(Kennan) (wkqwu)]

I also found it can happen in devstack in setup process for neutron as below:

2014-02-10 15:59:41 File "/usr/local/bin/neutron-db-manage", line 6, in <module>
2014-02-10 15:59:41 from neutron.db.migration.cli import main
2014-02-10 15:59:41 File "/opt/stack/neutron/neutron/db/migration/cli.py", line 24, in <module>
2014-02-10 15:59:41 from oslo.config import cfg
2014-02-10 15:59:41 ImportError: No module named config
2014-02-10 15:59:41 ++ failed

I just work- round the issue use following soft link

sudo ln -s /opt/stack/oslo.config/oslo/config/ /usr/local/lib/python2.7/dist-packages/oslo/config

anyway, not sure it is common issue as above

[Tracy Jones (tjones-i)]

it is happening to me too on a fresh /opt/stack/nova at TOT. The workaround you posted is not working for me. my path is correct. It almost looks like the python env used in tox is not the same as what i run at the command line

/opt/stack/nova/.tox/py27/bin/python
>>> from oslo.config import cfg
>>>

works fine. I'm kind of stuck here.

[Davanum Srinivas (DIMS) (dims-v)]

This should be fixed now. please reopen if needed:
https://github.com/openstack/nova/commit/01a56d64368564da89501e358d4244250528a675