having oslo.sphinx in namespace package causes issues with devstack

[Doug Hellmann]

http://lists.openstack.org/pipermail/openstack-dev/2014-January/023759.html

We've decided to rename oslo.sphinx to oslosphinx. This will require small changes in the doc builds for a lot of the other projects.

The problem seems to be when we pip install -e oslo.config on the system, then pip install oslo.sphinx in a venv. oslo.config is unavailable in the venv, apparently because the namespace package for o.s causes the egg-link for o.c to be ignored.

[Doug Hellmann (doug-hellmann)]

See https://review.openstack.org/#/q/I7788a9d6b5984fdfcc4678f2182104d2eb8a2be0,n,z for change series.

[Doug Hellmann (doug-hellmann)]

The git repository has been moved from openstack/oslo.sphinx to openstack/oslosphinx, so we're ready to work on updating consuming projects.

[Mark McLoughlin (markmc)]

Uggh, digging up the details behind this was pretty terrible ...

There is clearly some underlying issue we're papering over here ... but nowhere do I see any analysis of that underlying issue recorded for posterity

For example, in https://review.openstack.org/71589

  "Having the sphinx theme in the same namespace package
    as the production code has been causing issues with
    devstack installations."

The thread linked above ends with http://lists.openstack.org/pipermail/openstack-dev/2013-December/023002.html

 "Basically you can very easily get yourself in a situation where *just
   enough* of the distro package is left behind that pip thinks its there,
   so won't install it, but the python loader doesn't, so won't work.

   Then much sadness."

Or in https://review.openstack.org/#/c/65414/1//COMMIT_MSG

  "After merging https://review.openstack.org/64226 devstack gate started failing."

Oh, now I'm getting somewhere http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-01-08.log

   <dhellmann> fungi: the tl;dr is that when oslo.config is installed globally with "pip install -e" and oslo.sphinx is installed in a virtualenv with "pip install" python in the virtualenv can't find oslo.config
  ...
  <dhellmann> sdague: I think bnemec worked out that it's caused by a combination of pip install -e and something else in the same namespace package not being installed in that mode
  <dhellmann> sdague: alternatives to fix it seem to be install oslo.sphinx with pip install -e or change its package name
  <dhellmann> sdague: or don't install it at all, of course
  <bnemec> dhellmann: sdague: Right, the problem is that we pip install -e oslo.config in the system site packages, then pip install oslo.sphinx in the venv.
  <bnemec> That combination results in oslo.config being unavailable in the venv.
  <dhellmann> fungi: common namespace, split, *and* installed with .egg-link in some cases (I think)
  <dstufft> namespaces don't play well with -e
  <dhellmann> so for normal production oslo libs, we'd just install them from devstack with pip install -e and be done
  <dhellmann> but because oslo.sphinx is not a production lib, that's not necessarily the best answer
  <dhellmann> we should probably just reserve the oslo namespace for production libraries, and rename the theme package
  <dhellmann> well, it could also be said that if devstack didn't use that -e option this wouldn't be an issue, so it's a combination of all of it

And, at last, I find Ben's email ! http://lists.openstack.org/pipermail/openstack-dev/2014-January/023759.html

  The problem seems to be when we pip install -e oslo.config on the
  system, then pip install oslo.sphinx in a venv. oslo.config is
  unavailable in the venv, apparently because the namespace package for
  o.s causes the egg-link for o.c to be ignored. Pretty much every other
  combination I've tried (regular pip install of both, or pip install -e
  of both, regardless of where they are) works fine, but there seem to be
  other issues with all of the other options we've explored so far.

  We can't remove the pip install -e of oslo.config because it has to be
  used for gating...

[Doug Hellmann (doug-hellmann)]

Ceilometer change: https://review.openstack.org/#/c/73287/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72991
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c926216a9bd64d5cdd9731bd34e03613940d32e9
Submitter: Jenkins
Branch: master

commit c926216a9bd64d5cdd9731bd34e03613940d32e9
Author: Ben Nemec <email address hidden>
Date: Fri Feb 7 23:38:39 2014 +0000

    Replace oslo.sphinx with oslosphinx

    oslosphinx is now available as a replacement for oslo.sphinx that
    won't conflict with oslo.config in virtual envs.

    Change-Id: I7c116f816af895261e76af385ee3e9288e6fa70f
    Closes-Bug: #1277168

[Doug Hellmann (doug-hellmann)]

Dirk is working on patches for all the projects currently listed. I don't know why launchpad won't let me assign the ticket to him.

[Doug Hellmann (doug-hellmann)]

All changes are here: https://review.openstack.org/#/q/topic:oslosphinx,n,z

[Doug Hellmann (doug-hellmann)]

Andreas Jaeger is working on this for the CI docs in https://review.openstack.org/#/c/73709/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73278
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=de6d92fb924911aaf8bcae72995012b5e1781045
Submitter: Jenkins
Branch: master

commit de6d92fb924911aaf8bcae72995012b5e1781045
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 14:39:21 2014 +0100

    Switch over to oslosphinx

    Having the sphinx theme in the same namespace package
    as the production code has been causing issues with
    devstack installations. The solution settled on was
    to rename oslo.sphinx to oslosphinx because oslo.sphinx
    is not a "production" library.

    See the linked bug report for more background details.

    Closes-Bug: #1277168
    Change-Id: I220b8901cef36499e91b92719f1e8e5461a95e92

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73273
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=82ba4286e0ed3b6633ea9a97200fa6301776f4f7
Submitter: Jenkins
Branch: master

commit 82ba4286e0ed3b6633ea9a97200fa6301776f4f7
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 14:27:54 2014 +0100

    Switch over to oslosphinx

    oslosphinx is the new name of oslo.sphinx

    Closes-Bug: #1277168
    Change-Id: Ibfb9e59fbb3ef51a821b4f412601e26cb7f703f4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73282
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=69df886b761dd6ee47a15cad4f11e030f1e1c286
Submitter: Jenkins
Branch: master

commit 69df886b761dd6ee47a15cad4f11e030f1e1c286
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 14:44:32 2014 +0100

    Switch over to oslosphinx

    oslosphinx is the new name of oslo.sphinx

    Closes-Bug: #1277168
    Change-Id: I8aa08e3e16d46209b9aeb20a70b3984b2a4ae5bb

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73277
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=40898f1bd01caa17ad8fec9ec3887a042d8a0386
Submitter: Jenkins
Branch: master

commit 40898f1bd01caa17ad8fec9ec3887a042d8a0386
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 14:35:23 2014 +0100

    Switch over to oslosphinx

    oslosphinx is the new name of oslo.sphinx

    Closes-Bug: #1277168
    Change-Id: I9c52b7ad5459c51ed3d2a92d9011c34c9de4de5a

[Ben Nemec (bnemec)]

I think this is actually done for tripleo. I don't see any remaining references to oslo.sphinx in my devtest environment.

[Robert Collins (lifeless)]

@dhellman, whats needed for this in tripleo?

[Doug Hellmann (doug-hellmann)]

@lifeless - If you're building docs with sphinx using the OpenStack theme contained in oslo.sphinx, that should change to "oslosphinx" (removing the namespace package). I don't see oslo.sphinx being used in tripleo, so you might not need to do anything at all.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73367
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=e53f7792ad6156ad7b16a613dd86a477d5d537e3
Submitter: Jenkins
Branch: master

commit e53f7792ad6156ad7b16a613dd86a477d5d537e3
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 18:44:59 2014 +0100

    Switch over to oslosphinx

    oslosphinx is now available as a replacement for oslo.sphinx that
    won't conflict with oslo.config in virtual envs

    Change-Id: I7dc3349c8102cc8f358de7e0088128b6705c5f8e
    Closes-Bug: #1277168

[Doug Hellmann (doug-hellmann)]

tripleo is still using oslo.sphinx in their new os-cloud-config repository

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81187

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/81189

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73280
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=5f5641482c4c841a657daa66752fbab71de3308a
Submitter: Jenkins
Branch: master

commit 5f5641482c4c841a657daa66752fbab71de3308a
Author: Dirk Mueller <email address hidden>
Date: Thu Feb 13 14:41:55 2014 +0100

    Switch over to oslosphinx

    oslosphinx is the new name of oslo.sphinx

    Closes-Bug: #1277168
    Change-Id: I717c92bf62b8820517d1c0f1f83c75713a680ed7

[Sergey Lukjanov (slukjanov)]

Fixed in cinder https://github.com/openstack/cinder/commit/ea7d4a599224b5d0c7674d03993bbe72c49f0d51

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/83042

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/83042
Committed: https://git.openstack.org/cgit/openstack/marconi/commit/?id=733e6622027900e424dff83737b7723431c273f9
Submitter: Jenkins
Branch: master

commit 733e6622027900e424dff83737b7723431c273f9
Author: Sergey Lukjanov <email address hidden>
Date: Wed Mar 26 15:24:15 2014 +0400

    Switch to oslosphinx

    It was renamed.

    Change-Id: I7ed8e5d62979020af2f26921e50efad9a3489410
    Closes-Bug: #1277168

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/79303
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=96df098becbc33ecc28a8ea89e352b443cb34a2e
Submitter: Jenkins
Branch: master

commit 96df098becbc33ecc28a8ea89e352b443cb34a2e
Author: Cyril Roelandt <email address hidden>
Date: Mon Mar 10 13:32:08 2014 +0100

    Switch over to oslosphinx

    oslosphinx is the new name of oslo.sphinx

    Closes-Bug: #1277168
    Change-Id: I4cebcffb36724d699147a26eb1132b122cfe669c

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/89282

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.