rewrite pull-[lp|debian|uca]-source

[Logan Rosen]

I've essentially re-written the pull-[lp|debian|uca]-source scripts, consolidating them all to use a common codebase and extending the functionality to allow pulling not only source but also debs, udebs, and ddebs.

The updates also allow specifying binary package names, which the original pull-*-source didn't work with.

This also fixes bug 1700846, where pull-debian-source fails for older packages where the public key that the DSC was signed with no longer is in the keyring.

This also allows actually getting ddebs for old packages - unlike ddebs.ubuntu.com, which only provides ddebs for the latest few package versions, pull-lp-ddebs will actually pull ddebs for all older package versions, meaning you can debug a problem without being forced to upgrade to the latest version of all relevant packages first.

This also fixes various flake8/pylint failures when building in bionic. The current version (without my patches) no longer builds in bionic due to updates to flake8 and pylint causing ubuntu-dev-tools build failures.

Original description:

Errors like the following are happening because Debian's DDE [1] is dead [2]:

logan@logan-VMware:~$ pull-debian-source d-rats
pull-debian-source: Error: Unable to retrieve package information from DDE: http://dde.debian.net/dde/q/udd/dist/d:debian/r:sid/p:d-rats/?t=json (<urlopen error [Errno -2] Name or service not known>)
pull-debian-source: Error: Unable to find d-rats in Debian suite "sid".

Maybe someone would be willing to host/maintain a new instance of DDE? Or is there another way to programmatically grab the Debian/Ubuntu source package name from the binary package name?

[1] https://wiki.debian.org/DDE
[2] https://lists.debian.org/debian-devel/2014/09/msg00603.html

Related bugs:
 * bug 1508948: pull-debian-source fails use of rmadison on wily

[Logan Rosen (logan)]

An alternative is to query the UDD public mirror [1] directly using Psycopg, but I'm not sure if this is ideal (it also would add another dependency/recommendation to ubuntu-dev-tools). Thoughts?

[1] http://public-udd-mirror.xvm.mit.edu/

[Colin Watson (cjwatson)]

You can get it from LP following a recent API extension. Something like this:

            try:
                bpph = archive.getBinaryPackage(package, release, pocket)
            except PackageNotFoundException, e:
                Logger.error(str(e))
                sys.exit(1)
            spph = bpph.build.getLatestSourcePublication()
            if spph is not None:
                package = spph.getPackageName()

I'll try to finish off my branch to convert to this shortly; I hadn't realised it was going to be urgent.

[Iain Lane (laney)]

Any news or should someone else pick this up?

[Scott Moser (smoser)]

in wily, pull-lp-source works for me, but a simple 'pull-debian-source hello' would fail.
The change in behavior from the working vivid for my specific use case is that rmadison has changed and is hitting a different server.

on wily:
$ rmadison -u debian -s sid -a source hello
hello | 2.10-1 | unstable | source

on trusty (and vivid)
$ rmadison -u debian -s sid -a source hello
 hello | 2.10-1 | sid | source

The easiest fix for me to make 'pull-debian-source hello' work again was this:
=== modified file 'ubuntutools/archive.py'
--- ubuntutools/archive.py 2014-12-18 23:18:00 +0000
+++ ubuntutools/archive.py 2015-10-21 20:59:55 +0000
@@ -604,6 +604,8 @@
 def rmadison(url, package, suite=None, arch=None):
     "Call rmadison and parse the result"
     cmd = ['rmadison', '-u', url]
+ if suite == "sid":
+ suite = "unstable"
     if suite:
         cmd += ['-s', suite]
     if arch:

That fixes my itch at least.

[Hans Joachim Desserud (hjd)]

Now that bug 1508948 is fixed, pull-debian-source seems to be working again on Ubuntu Xenial. As said, that adresses the rmadison failure, so I don't know whether this bug should be left open or what happens to it?

[Adam Conrad (adconrad)]

This bug is still relevant for binary->source mapping.

[Dan Streetman (ddstreet)]

debdiff containing patch to zesty pull-lp-source. This applies to T/X/Y pull-lp-source also.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1453330-zesty.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Khurshid Alam (khurshid-alam)]

Please fix it for zesty. It is preventing users from submitting patches for zesty.

[Dan Streetman (ddstreet)]

https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/lp1453330

[Khurshid Alam (khurshid-alam)]

Is this going to land for zesty or not?

[Logan Rosen (logan)]

Dan, is that branch working? If so, can you please propose it for merging?

[Dan Streetman (ddstreet)]

Merge proposal created:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/lp1453330/+merge/319865

[Dan Streetman (ddstreet)]

Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

[Dmitry Shachnev (mitya57)]

> Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

Sorry, I do not know the code much, so I would prefer someone else to review it. There are some other people with rights to merge subscribed here :)

[Dan Streetman (ddstreet)]

Iain, you created pull-lp-source, can you review this merge request?

[Iain Lane (laney)]

Ok, but you should look at the thing that I created and compare it to today's pull-lp-source. :)

[Khurshid Alam (khurshid-alam)]

I can confirm: the patch by Dan is woking!

[Michael Terry (mterry)]

The linked MP is marked as merged, but I don't think it is, looking at trunk. Dan, can you get your MP sorted and reviewable, and then subscribe sponsors again?

[Dan Streetman (ddstreet)]

Yep, I just changed it to work-in-progress, until i finish updating it so pull-debian-* works also.

[Dan Streetman (ddstreet)]

After quite a bit of major updating, I have some test pkgs built here:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330/+packages

it moves pull-lp-source to 'pull-pkg'; that is able to pull from debian, ubuntu, or uca, and it can pull source, binaries, ddebs, or di files. It can also simply list all a package's files.

There are small bash scripts to help calling pull-pkg:
pull-(lp|debian|uca)-(source|binaries|ddebs|di)

can anyone interested in this test out the updated scripts from the test package please? I'd like input before updating the merge request.

My changes are all in my git repo in the branch 'pull-lp':
https://code.launchpad.net/~ddstreet/+git/ubuntu-dev-tools/+ref/pull-lp

[Dan Streetman (ddstreet)]

Ok, I updated my merge request to a git-based merge request:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

also, I consolidated the script naming; so it's now:

pull-lp-source
pull-lp-debs
pull-lp-ddebs
pull-lp-udebs

which I think is more obvious, as to what you want to pull (instead of pull-lp-binaries and pull-lp-di).

And the renamed 'main' script is:

pull-pkg

laney, mterry, any chance you could review the patches for possible merging? I know it's quite a lot of code, but it does add a lot of useful functionality, I think.

[Dan Streetman (ddstreet)]

Also, I have pkgs built for xenial/yakkety/zesty with the latest merge proposal code, in this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330

[Dan Streetman (ddstreet)]

After quite a bit of patch editing and git rebasing of my working repo, I've re-opened the merge request. Any coredevs who are interested please review the patches if you have time:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

[Dan Streetman (ddstreet)]

Sponsors, @mterry, @laney, with 17.10 release behind us, does anyone have time to review this merge request?

[Dan Streetman (ddstreet)]

@mapreri you've uploaded several of the most recent ubuntu-dev-tools changes, can you review this merge request?

[Mattia Rizzolo (mapreri)]

Most probably I can, yes. But it's not a quick thing, I need to find some appropriate time to review.

BTW, even after merging some thing I'd prefer if you could stick around and deal with upcoming bugs in pull-pkg in the next months.

[Dan Streetman (ddstreet)]

> But it's not a quick thing, I need to find some appropriate time to review.

absolutely, I'm hoping it can get merged by b release next year, but it's obviously not critical.

> BTW, even after merging some thing I'd prefer if you could stick around and deal with
> upcoming bugs in pull-pkg in the next months.

yes of course, I'm not going anywhere.

thanks!

[Dan Streetman (ddstreet)]

@mapreri, just checking if you have had any time for any review of this yet.

[Dan Streetman (ddstreet)]

@dbrung can you please review/merge this when you have a chance.

I checked with flake8 and pylint, and fixed the couple flake8 errors. pylint shows quite a few 'errors' but most are with the existing ubuntu-dev-tools codebase, not my additions; such as 'missing-docstring' which I considered harmless and out of scope to fix in my patchset.

FYI, this latest patchset has been waiting on review for 4+ months.

[Dan Streetman (ddstreet)]

Logen Rosen,

I have kind-of hijacked your bug report here, and my changes go far beyond fixing just what is reported in this bug description; do you mind if I update the bug subject and description to reflect the more widespread changes in my current merge request?

[Dan Streetman (ddstreet)]

Hi all,

ubuntu-dev-tools maintainers (@bdrung and @stefanor) haven't responded to my irc pings or emails for months now, I assume they consider my changes too large to find time to review (but I don't know, since they aren't replying to me).

ubuntu foundations/server teams don't see any value to my changes and so are similarly ignoring this.

So I've just forked ubuntu-dev-tools and will maintain my own fork with all its improvements at this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/ubuntu-dev-tools

I'll keep it up to date with whatever changes upstream (very little recently), and keep adding new features and functions for anyone interested.

[Mattia Rizzolo (mapreri)]

Well, now that bionic is done, and buster has still about an year to come, let me just merge it and have you deal with whatever happens.. :)

[Dan Streetman (ddstreet)]

> Well, now that bionic is done, and buster has still about an year to come, let me just merge
> it and have you deal with whatever happens.. :)

All my commits are now in this branch:

https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+ref/fork

I'm actively working on it so I may push -f or rebase, so it might be hard to review; I would not recommend trying to use it for review or merging it (assuming you were serious). I got tired of waiting (many months) while I wanted to continue developing it, so i stopped waiting.

[Mattia Rizzolo (mapreri)]

I am serious: you are right on being tired of waiting, so I'm asking you to give me something I can straight merge and upload as-is to unstable.

It's clear I'm not going to review it as I hoped I would, so there is little point in trying to defer it any more (and I kind of doubt you'll find many other people willing to seriously review that).

[Simon Quigley (tsimonq2)]

Unsubscribing sponsors for now; ping mapreri and ddstreet.

[Dan Streetman (ddstreet)]

merged and released in focal version 0.175