wishlist: parseable dependency list

[Thomas Goirand]

Hi guys!

First of all, thanks so much for working on PBR, which is really a wonderful software for package maintainers. So far, I just love it.

Robert Collins told me that requirements.txt and test-requirements.txt are going away, being integrated in setup.cfg. I don't mind if it does, but what I would need then, would be an easy way to extract the dependencies from setup.cfg. Generating the egg-info is *not* what I want, as I don't want any file to be written when designing a Debian package.

For example, what would be great would be something like:

pbr extract-runtime-depends

and

pbr extract-test-depends

that would just output on stdout a list of package and their required version. Large amount of bonus points if it can only tell me about the lower bound of the dependencies (as this is practically the only information which should be wired in the debian/control file).

My end result would be something like this, which I currently already produce with "pkgos-parse-requirements" (included in the recent versions of openstack-pkg-tools package in both Debian and Ubuntu):

Build-Depends: debhelper (>= 9),
               dh-python,
               dh-systemd,
               openstack-pkg-tools (>= 23~),
               po-debconf,
               python-all,
               python-pbr,
               python-setuptools,
               python-sphinx,
Build-Depends-Indep: python-babel,
                     python-barbicanclient (>= 3.0.1),
                     python-boto,
                     python-cinderclient (>= 1:1.1.0),
                     python-coverage,
                     python-decorator,
                     python-eventlet (>= 0.16.1),
                     python-fixtures,
                     python-glanceclient (>= 1:0.15.0),
                     python-greenlet,
                     python-hacking,
                     python-ironicclient (>= 0.2.1),
                     python-iso8601,
                     python-jinja2,
                     python-jsonschema,
                     python-keystonemiddleware (>= 1.5.0),
                     python-lxml,
                     python-migrate (>= 0.9.5),
                     python-mock,
                     python-mox3,
                     python-mysqldb,
                     python-netaddr,
                     python-neutronclient (>= 2.3.11),
                     python-oslo.concurrency (>= 1.8.0),
                     python-oslo.config (>= 1:1.9.3),
                     python-oslo.context (>= 0.2.0),
                     python-oslo.db (>= 1.7.0),
                     python-oslo.i18n (>= 1.5.0),
                     python-oslo.log (>= 1.0.0),
                     python-oslo.messaging (>= 1.8.0),
                     python-oslo.middleware (>= 1.0.0),
                     python-oslo.rootwrap (>= 1.6.0),
                     python-oslo.serialization (>= 1.4.0),
                     python-oslo.utils (>= 1.4.0),
                     python-oslo.vmware (>= 0.11.1),
                     python-oslosphinx (>= 2.5.0),
                     python-oslotest (>= 1.5.1),
                     python-paramiko,
                     python-paste,
                     python-pastedeploy,
                     python-psutil,
                     python-psycopg2,
                     python-pyasn1,
                     python-requests-mock (>= 0.6.0),
                     python-rfc3986,
                     python-routes,
                     python-six (>= 1.9.0),
                     python-sqlalchemy,
                     python-stevedore (>= 1.3.0),
                     python-suds,
                     python-tempest-lib (>= 0.4.0),
                     python-testtools,
                     python-webob,
                     subunit,
                     testrepository,
                     websockify,
Depends: python-babel,
         python-boto,
         python-cinderclient (>= 1:1.1.0),
         python-decorator,
         python-eventlet (>= 0.16.1),
         python-glanceclient (>= 1:0.15.0),
         python-greenlet,
         python-iso8601,
         python-jinja2,
         python-jsonschema,
         python-keystonemiddleware (>= 1.5.0),
         python-lxml,
         python-migrate (>= 0.9.5),
         python-netaddr,
         python-neutronclient (>= 2.3.11),
         python-oslo.concurrency (>= 1.8.0),
         python-oslo.config (>= 1:1.9.3),
         python-oslo.context (>= 0.2.0),
         python-oslo.db (>= 1.7.0),
         python-oslo.i18n (>= 1.5.0),
         python-oslo.log (>= 1.0.0),
         python-oslo.messaging (>= 1.8.0),
         python-oslo.middleware (>= 1.0.0),
         python-oslo.rootwrap (>= 1.6.0),
         python-oslo.serialization (>= 1.4.0),
         python-oslo.utils (>= 1.4.0),
         python-paramiko,
         python-paste,
         python-pastedeploy,
         python-psutil,
         python-pyasn1,
         python-rfc3986,
         python-routes,
         python-six (>= 1.9.0),
         python-sqlalchemy,
         python-stevedore (>= 1.3.0),
         python-webob,
         websockify,
         ${misc:Depends},
         ${python:Depends},

I can of course parse and rework the output of pbr with such proposed new options instead of the currently existing requirements.txt. Though the thing which I don't want is too much hacking extracting the needed info from setup.cfg so that it produces what I have already right now through the requirements.txt.

Just to make it clear: this currently is only helpers for me, as a package maintainer, and this is not (yet) a fully automated tool. However, I may some day implement this as a lintian check (this is on my todo list), so probably it'd be a good idea to have it written in a not-so-dirty way.

[Robert Collins (lifeless)]

So, considerations:
 - that list isn't python version aware, let alone the full capabilities of markers.
 - egg_info is the standard interchange for python packaging metadata today. I don't see much value in building a different format which would need to be parsed separately.
 - as far as files go, just running setup.py *anything* is likely to create a file due to easy_install. We want to fix that, but thats where it is today.

[Thomas Goirand (thomas-goirand)]

My reply to that...

The list above is what I want as a final result from *my* tooling. But of course, I do not expect pbr to produce that directly. I just wish it made some kind of output that I could easily transform into the above. I can be what's in the *.egg-info/requires.txt, that's fine, as long as it doesn't actually write any file.

It doesn't have to be done through "python setup.py", it could be a pbr command line thing.

What's important here, is that I get a way to extract the information from setup.cfg, if it moves there. Just something that would do that from pbr could be enough, then I'll do the rest using my ugly (but useful and not for production) shell scripts! :)