buildout claims to have worked but testrunner cannot import lazr.uri (a dependency)

[Jonathan Lange]

After running:
 $ python bootstrap.py
 $ ./bin/buildout

Then running the tests fails with:
 $ ./bin/test
Test-module import failures:

Module: lazr.restfulclient.tests.test_docs

ImportError: No module named uri

Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
  Ran 16 tests with 0 failures and 0 errors in 0.002 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

Test-modules with import problems:
  lazr.restfulclient.tests.test_docs

This appears to be because lazr.uri does not import:

$ ./bin/py
Python 2.7.2+ (default, Aug 16 2011, 07:24:41)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lazr.uri
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named uri
>>>

[Jonathan Lange (jml)]

Investigation shows that lazr.uri isn't listed as a dependency in setup.py, and that even when you do add it, it won't import, probably due to Python namespace issues.

Note that lazr.uri is installed on my system as an Ubuntu package at exactly the same version that versions.cfg requests.

[Benji York (benji)]

Using a clean Python (i.e., non-system, no installed packages) or a
virtualenv will solve the above problem. This is generally the way
buildout should be used unless the system packages are closely managed
as part of the development environment.

I'll add comments to this effect to the HACKING document.

[Robert Collins (lifeless)]

Our platform is Python on Ubuntu, not virtualenv on python on Ubuntu,
so this not working is a pretty big deal, not an expected situation to
avoid.

[Benji York (benji)]

On Tue, Sep 13, 2011 at 4:27 PM, Robert Collins
<email address hidden> wrote:
> Our platform is Python on Ubuntu, not virtualenv on python on Ubuntu,
> so this not working is a pretty big deal, not an expected situation to
> avoid.

To be sure I'm being clear: I'm talking about development not usage.

Unfortunately given the limitations of Python's module system I don't
know of a way to avoid conflicts with system-installed packages like
this bug represents (or not, see below).

Addressing the system Python problem was the intent of Gary's work on
buildout 1.5. Unfortunately that work hasn't found the acceptance we
had hoped for and is being removed in buildout 2.0.

There is some indication that Python 3 will grow the ability to handle
the situation better, but that's not happened yet and of course we're
not on 3 yet.

All that being said, after trying to reproduce the problem, I can't. It
looks like lazr.uri should be listed in setup.py as a dependency, but
that shouldn't have caused this problem because it's apparently a
transitive dependency already.

Jonathan: if you start Python without the bin/py wrapper, can you import
lazr.uri?
--
Benji York

[Jonathan Lange (jml)]

On Wed, Sep 14, 2011 at 2:22 PM, Benji York <email address hidden> wrote:
...
> All that being said, after trying to reproduce the problem, I can't.  It
> looks like lazr.uri should be listed in setup.py as a dependency, but
> that shouldn't have caused this problem because it's apparently a
> transitive dependency already.
>
> Jonathan: if you start Python without the bin/py wrapper, can you import
> lazr.uri?

Yes.

jml

[Robert Collins (lifeless)]

On Thu, Sep 15, 2011 at 1:22 AM, Benji York <email address hidden> wrote:
> On Tue, Sep 13, 2011 at 4:27 PM, Robert Collins
> <email address hidden> wrote:
>> Our platform is Python on Ubuntu, not virtualenv on python on Ubuntu,
>> so this not working is a pretty big deal, not an expected situation to
>> avoid.
>
> To be sure I'm being clear: I'm talking about development not usage.
>
> Unfortunately given the limitations of Python's module system I don't
> know of a way to avoid conflicts with system-installed packages like
> this bug represents (or not, see below).
>
> Addressing the system Python problem was the intent of Gary's work on
> buildout 1.5.  Unfortunately that work hasn't found the acceptance we
> had hoped for and is being removed in buildout 2.0.

Yup, I'm talking development too - we deploy LP as we develop it
though (for a variety of good reasons but mainly to avoid last minute
surprises). So things that are worrying in dev... are worrying in prod
:)

For instance, we can't guarantee that some other packaged bit of
software won't drag lazr.uri in on a prod server. If that would break
our other deployed code, that would be a nasty surprise!

-Rob

[Martin Pool (mbp)]

This might be related to bug 796992.

[Benji York (benji)]

On Wed, Sep 14, 2011 at 3:41 PM, Robert Collins
<email address hidden> wrote:
> On Thu, Sep 15, 2011 at 1:22 AM, Benji York <email address hidden> wrote:
>> On Tue, Sep 13, 2011 at 4:27 PM, Robert Collins
>> <email address hidden> wrote:
>>> Our platform is Python on Ubuntu, not virtualenv on python on Ubuntu,
>>> so this not working is a pretty big deal, not an expected situation to
>>> avoid.
>>
>> To be sure I'm being clear: I'm talking about development not usage.
>>
>> Unfortunately given the limitations of Python's module system I don't
>> know of a way to avoid conflicts with system-installed packages like
>> this bug represents (or not, see below).
>>
>> Addressing the system Python problem was the intent of Gary's work on
>> buildout 1.5.  Unfortunately that work hasn't found the acceptance we
>> had hoped for and is being removed in buildout 2.0.
>
> Yup, I'm talking development too - we deploy LP as we develop it
> though (for a variety of good reasons but mainly to avoid last minute
> surprises). So things that are worrying in dev... are worrying in prod
> :)

Ah, now I see where you're coming from. You are thinking in the context
of LP, while I was thinking of developing the individual package.

> For instance, we can't guarantee that some other packaged bit of
> software won't drag lazr.uri in on a prod server. If that would break
> our other deployed code, that would be a nasty surprise!

Indeed. Given a context of LP that's certainly something to worry
about. I /think/ the best way to deal with that is to treat the
individual Python projects (lazr.*) as isolated things when developing
them (i.e., using a clean Python or virtualenv) and treat LP as a part
of a larger whole that includes those packages and the system it is
running on top of.

I think your point -- and a good one it is -- is that our best bet is
for our LP development environments reflect the production systems as
best as is practical.

(I'm getting far afield of this bug now. I wonder if I should have
replied to the list instead.)
--
Benji York