rewrite pull-[lp|debian|uca]-source

Bug #1453330 reported by Logan Rosen
62
This bug affects 7 people
Affects Status Importance Assigned to Milestone
ubuntu-dev-tools (Debian)
Fix Released
Unknown
ubuntu-dev-tools (Ubuntu)
Fix Released
Wishlist
Unassigned

Bug Description

I've essentially re-written the pull-[lp|debian|uca]-source scripts, consolidating them all to use a common codebase and extending the functionality to allow pulling not only source but also debs, udebs, and ddebs.

The updates also allow specifying binary package names, which the original pull-*-source didn't work with.

This also fixes bug 1700846, where pull-debian-source fails for older packages where the public key that the DSC was signed with no longer is in the keyring.

This also allows actually getting ddebs for old packages - unlike ddebs.ubuntu.com, which only provides ddebs for the latest few package versions, pull-lp-ddebs will actually pull ddebs for all older package versions, meaning you can debug a problem without being forced to upgrade to the latest version of all relevant packages first.

This also fixes various flake8/pylint failures when building in bionic. The current version (without my patches) no longer builds in bionic due to updates to flake8 and pylint causing ubuntu-dev-tools build failures.

Original description:

Errors like the following are happening because Debian's DDE [1] is dead [2]:

logan@logan-VMware:~$ pull-debian-source d-rats
pull-debian-source: Error: Unable to retrieve package information from DDE: http://dde.debian.net/dde/q/udd/dist/d:debian/r:sid/p:d-rats/?t=json (<urlopen error [Errno -2] Name or service not known>)
pull-debian-source: Error: Unable to find d-rats in Debian suite "sid".

Maybe someone would be willing to host/maintain a new instance of DDE? Or is there another way to programmatically grab the Debian/Ubuntu source package name from the binary package name?

[1] https://wiki.debian.org/DDE
[2] https://lists.debian.org/debian-devel/2014/09/msg00603.html

Related bugs:
 * bug 1508948: pull-debian-source fails use of rmadison on wily

Tags: patch
Logan Rosen (logan)
summary: - pull-debian-source not getting versions because DDE is dead
+ pull-debian-source not getting source for binary because DDE is dead
description: updated
Logan Rosen (logan)
summary: - pull-debian-source not getting source for binary because DDE is dead
+ pull-{lp,debian}-source not getting source for binary because DDE is
+ dead
description: updated
Revision history for this message
Logan Rosen (logan) wrote : Re: pull-{lp,debian}-source not getting source for binary because DDE is dead

An alternative is to query the UDD public mirror [1] directly using Psycopg, but I'm not sure if this is ideal (it also would add another dependency/recommendation to ubuntu-dev-tools). Thoughts?

[1] http://public-udd-mirror.xvm.mit.edu/

Revision history for this message
Colin Watson (cjwatson) wrote :

You can get it from LP following a recent API extension. Something like this:

            try:
                bpph = archive.getBinaryPackage(package, release, pocket)
            except PackageNotFoundException, e:
                Logger.error(str(e))
                sys.exit(1)
            spph = bpph.build.getLatestSourcePublication()
            if spph is not None:
                package = spph.getPackageName()

I'll try to finish off my branch to convert to this shortly; I hadn't realised it was going to be urgent.

Revision history for this message
Iain Lane (laney) wrote :

Any news or should someone else pick this up?

Changed in ubuntu-dev-tools (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Scott Moser (smoser) wrote :

in wily, pull-lp-source works for me, but a simple 'pull-debian-source hello' would fail.
The change in behavior from the working vivid for my specific use case is that rmadison has changed and is hitting a different server.

on wily:
$ rmadison -u debian -s sid -a source hello
hello | 2.10-1 | unstable | source

on trusty (and vivid)
$ rmadison -u debian -s sid -a source hello
 hello | 2.10-1 | sid | source

The easiest fix for me to make 'pull-debian-source hello' work again was this:
=== modified file 'ubuntutools/archive.py'
--- ubuntutools/archive.py 2014-12-18 23:18:00 +0000
+++ ubuntutools/archive.py 2015-10-21 20:59:55 +0000
@@ -604,6 +604,8 @@
 def rmadison(url, package, suite=None, arch=None):
     "Call rmadison and parse the result"
     cmd = ['rmadison', '-u', url]
+ if suite == "sid":
+ suite = "unstable"
     if suite:
         cmd += ['-s', suite]
     if arch:

That fixes my itch at least.

Scott Moser (smoser)
description: updated
Revision history for this message
Hans Joachim Desserud (hjd) wrote :

Now that bug 1508948 is fixed, pull-debian-source seems to be working again on Ubuntu Xenial. As said, that adresses the rmadison failure, so I don't know whether this bug should be left open or what happens to it?

Revision history for this message
Adam Conrad (adconrad) wrote :

This bug is still relevant for binary->source mapping.

Revision history for this message
Dan Streetman (ddstreet) wrote :

debdiff containing patch to zesty pull-lp-source. This applies to T/X/Y pull-lp-source also.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1453330-zesty.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Changed in ubuntu-dev-tools (Debian):
status: Unknown → New
Revision history for this message
Khurshid Alam (khurshid-alam) wrote :

Please fix it for zesty. It is preventing users from submitting patches for zesty.

Revision history for this message
Dan Streetman (ddstreet) wrote :
Revision history for this message
Khurshid Alam (khurshid-alam) wrote :

Is this going to land for zesty or not?

Revision history for this message
Logan Rosen (logan) wrote :

Dan, is that branch working? If so, can you please propose it for merging?

Revision history for this message
Dan Streetman (ddstreet) wrote :
Revision history for this message
Dan Streetman (ddstreet) wrote :

Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

tags: added: sts-sponsor
Revision history for this message
Dmitry Shachnev (mitya57) wrote :

> Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

Sorry, I do not know the code much, so I would prefer someone else to review it. There are some other people with rights to merge subscribed here :)

Dan Streetman (ddstreet)
tags: removed: sts-sponsor
Revision history for this message
Dan Streetman (ddstreet) wrote :

Iain, you created pull-lp-source, can you review this merge request?

Revision history for this message
Iain Lane (laney) wrote :

Ok, but you should look at the thing that I created and compare it to today's pull-lp-source. :)

Revision history for this message
Khurshid Alam (khurshid-alam) wrote :

I can confirm: the patch by Dan is woking!

Revision history for this message
Michael Terry (mterry) wrote :

The linked MP is marked as merged, but I don't think it is, looking at trunk. Dan, can you get your MP sorted and reviewable, and then subscribe sponsors again?

Revision history for this message
Dan Streetman (ddstreet) wrote :

Yep, I just changed it to work-in-progress, until i finish updating it so pull-debian-* works also.

Revision history for this message
Dan Streetman (ddstreet) wrote :

After quite a bit of major updating, I have some test pkgs built here:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330/+packages

it moves pull-lp-source to 'pull-pkg'; that is able to pull from debian, ubuntu, or uca, and it can pull source, binaries, ddebs, or di files. It can also simply list all a package's files.

There are small bash scripts to help calling pull-pkg:
pull-(lp|debian|uca)-(source|binaries|ddebs|di)

can anyone interested in this test out the updated scripts from the test package please? I'd like input before updating the merge request.

My changes are all in my git repo in the branch 'pull-lp':
https://code.launchpad.net/~ddstreet/+git/ubuntu-dev-tools/+ref/pull-lp

Changed in ubuntu-dev-tools (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
Revision history for this message
Dan Streetman (ddstreet) wrote :

Ok, I updated my merge request to a git-based merge request:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

also, I consolidated the script naming; so it's now:

pull-lp-source
pull-lp-debs
pull-lp-ddebs
pull-lp-udebs

which I think is more obvious, as to what you want to pull (instead of pull-lp-binaries and pull-lp-di).

And the renamed 'main' script is:

pull-pkg

laney, mterry, any chance you could review the patches for possible merging? I know it's quite a lot of code, but it does add a lot of useful functionality, I think.

Revision history for this message
Dan Streetman (ddstreet) wrote :

Also, I have pkgs built for xenial/yakkety/zesty with the latest merge proposal code, in this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330

Revision history for this message
Dan Streetman (ddstreet) wrote :

After quite a bit of patch editing and git rebasing of my working repo, I've re-opened the merge request. Any coredevs who are interested please review the patches if you have time:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

Revision history for this message
Dan Streetman (ddstreet) wrote :

Sponsors, @mterry, @laney, with 17.10 release behind us, does anyone have time to review this merge request?

Revision history for this message
Dan Streetman (ddstreet) wrote :

@mapreri you've uploaded several of the most recent ubuntu-dev-tools changes, can you review this merge request?

Revision history for this message
Mattia Rizzolo (mapreri) wrote :

Most probably I can, yes. But it's not a quick thing, I need to find some appropriate time to review.

BTW, even after merging some thing I'd prefer if you could stick around and deal with upcoming bugs in pull-pkg in the next months.

Revision history for this message
Dan Streetman (ddstreet) wrote :

> But it's not a quick thing, I need to find some appropriate time to review.

absolutely, I'm hoping it can get merged by b release next year, but it's obviously not critical.

> BTW, even after merging some thing I'd prefer if you could stick around and deal with
> upcoming bugs in pull-pkg in the next months.

yes of course, I'm not going anywhere.

thanks!

Revision history for this message
Dan Streetman (ddstreet) wrote :

@mapreri, just checking if you have had any time for any review of this yet.

Revision history for this message
Dan Streetman (ddstreet) wrote :

@dbrung can you please review/merge this when you have a chance.

I checked with flake8 and pylint, and fixed the couple flake8 errors. pylint shows quite a few 'errors' but most are with the existing ubuntu-dev-tools codebase, not my additions; such as 'missing-docstring' which I considered harmless and out of scope to fix in my patchset.

FYI, this latest patchset has been waiting on review for 4+ months.

Revision history for this message
Dan Streetman (ddstreet) wrote :

Logen Rosen,

I have kind-of hijacked your bug report here, and my changes go far beyond fixing just what is reported in this bug description; do you mind if I update the bug subject and description to reflect the more widespread changes in my current merge request?

Dan Streetman (ddstreet)
summary: - pull-{lp,debian}-source not getting source for binary because DDE is
- dead
+ rewrite pull-[lp|debian]-source
Dan Streetman (ddstreet)
description: updated
summary: - rewrite pull-[lp|debian]-source
+ rewrite pull-[lp|debian|uca]-source
Dan Streetman (ddstreet)
description: updated
Dan Streetman (ddstreet)
description: updated
Dan Streetman (ddstreet)
description: updated
Changed in ubuntu-dev-tools (Ubuntu):
status: Triaged → In Progress
api.ng (hektve)
Changed in ubuntu-dev-tools (Ubuntu):
assignee: Dan Streetman (ddstreet) → HECTOR DAVID (hektve)
Colin Watson (cjwatson)
Changed in ubuntu-dev-tools (Ubuntu):
assignee: HECTOR DAVID (hektve) → Dan Streetman (ddstreet)
Revision history for this message
Dan Streetman (ddstreet) wrote :

Hi all,

ubuntu-dev-tools maintainers (@bdrung and @stefanor) haven't responded to my irc pings or emails for months now, I assume they consider my changes too large to find time to review (but I don't know, since they aren't replying to me).

ubuntu foundations/server teams don't see any value to my changes and so are similarly ignoring this.

So I've just forked ubuntu-dev-tools and will maintain my own fork with all its improvements at this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/ubuntu-dev-tools

I'll keep it up to date with whatever changes upstream (very little recently), and keep adding new features and functions for anyone interested.

Changed in ubuntu-dev-tools (Ubuntu):
assignee: Dan Streetman (ddstreet) → nobody
Revision history for this message
Mattia Rizzolo (mapreri) wrote :

Well, now that bionic is done, and buster has still about an year to come, let me just merge it and have you deal with whatever happens.. :)

Changed in ubuntu-dev-tools (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
importance: High → Wishlist
Revision history for this message
Dan Streetman (ddstreet) wrote :

> Well, now that bionic is done, and buster has still about an year to come, let me just merge
> it and have you deal with whatever happens.. :)

All my commits are now in this branch:

https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+ref/fork

I'm actively working on it so I may push -f or rebase, so it might be hard to review; I would not recommend trying to use it for review or merging it (assuming you were serious). I got tired of waiting (many months) while I wanted to continue developing it, so i stopped waiting.

Changed in ubuntu-dev-tools (Ubuntu):
assignee: Dan Streetman (ddstreet) → nobody
status: In Progress → Incomplete
Revision history for this message
Mattia Rizzolo (mapreri) wrote :

I am serious: you are right on being tired of waiting, so I'm asking you to give me something I can straight merge and upload as-is to unstable.

It's clear I'm not going to review it as I hoped I would, so there is little point in trying to defer it any more (and I kind of doubt you'll find many other people willing to seriously review that).

Revision history for this message
Simon Quigley (tsimonq2) wrote :

Unsubscribing sponsors for now; ping mapreri and ddstreet.

Revision history for this message
Dan Streetman (ddstreet) wrote :

merged and released in focal version 0.175

Changed in ubuntu-dev-tools (Ubuntu):
status: Incomplete → Fix Released
Changed in ubuntu-dev-tools (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.