pull-{lp,debian}-source not getting source for binary because DDE is dead

[Logan Rosen]

Errors like the following are happening because Debian's DDE [1] is dead [2]:

logan@logan-VMware:~$ pull-debian-source d-rats
pull-debian-source: Error: Unable to retrieve package information from DDE: http://dde.debian.net/dde/q/udd/dist/d:debian/r:sid/p:d-rats/?t=json (<urlopen error [Errno -2] Name or service not known>)
pull-debian-source: Error: Unable to find d-rats in Debian suite "sid".

Maybe someone would be willing to host/maintain a new instance of DDE? Or is there another way to programmatically grab the Debian/Ubuntu source package name from the binary package name?

[1] https://wiki.debian.org/DDE
[2] https://lists.debian.org/debian-devel/2014/09/msg00603.html

Related bugs:
 * bug 1508948: pull-debian-source fails use of rmadison on wily

[Logan Rosen (logan)]

An alternative is to query the UDD public mirror [1] directly using Psycopg, but I'm not sure if this is ideal (it also would add another dependency/recommendation to ubuntu-dev-tools). Thoughts?

[1] http://public-udd-mirror.xvm.mit.edu/

[Colin Watson (cjwatson)]

You can get it from LP following a recent API extension. Something like this:

            try:
                bpph = archive.getBinaryPackage(package, release, pocket)
            except PackageNotFoundException, e:
                Logger.error(str(e))
                sys.exit(1)
            spph = bpph.build.getLatestSourcePublication()
            if spph is not None:
                package = spph.getPackageName()

I'll try to finish off my branch to convert to this shortly; I hadn't realised it was going to be urgent.

[Iain Lane (laney)]

Any news or should someone else pick this up?

[Scott Moser (smoser)]

in wily, pull-lp-source works for me, but a simple 'pull-debian-source hello' would fail.
The change in behavior from the working vivid for my specific use case is that rmadison has changed and is hitting a different server.

on wily:
$ rmadison -u debian -s sid -a source hello
hello | 2.10-1 | unstable | source

on trusty (and vivid)
$ rmadison -u debian -s sid -a source hello
 hello | 2.10-1 | sid | source

The easiest fix for me to make 'pull-debian-source hello' work again was this:
=== modified file 'ubuntutools/archive.py'
--- ubuntutools/archive.py 2014-12-18 23:18:00 +0000
+++ ubuntutools/archive.py 2015-10-21 20:59:55 +0000
@@ -604,6 +604,8 @@
 def rmadison(url, package, suite=None, arch=None):
     "Call rmadison and parse the result"
     cmd = ['rmadison', '-u', url]
+ if suite == "sid":
+ suite = "unstable"
     if suite:
         cmd += ['-s', suite]
     if arch:

That fixes my itch at least.

[Hans Joachim Desserud (hjd)]

Now that bug 1508948 is fixed, pull-debian-source seems to be working again on Ubuntu Xenial. As said, that adresses the rmadison failure, so I don't know whether this bug should be left open or what happens to it?

[Adam Conrad (adconrad)]

This bug is still relevant for binary->source mapping.

[Dan Streetman (ddstreet)]

debdiff containing patch to zesty pull-lp-source. This applies to T/X/Y pull-lp-source also.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1453330-zesty.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Khurshid Alam (khurshid-alam)]

Please fix it for zesty. It is preventing users from submitting patches for zesty.

[Dan Streetman (ddstreet)]

https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/lp1453330

[Khurshid Alam (khurshid-alam)]

Is this going to land for zesty or not?

[Logan Rosen (logan)]

Dan, is that branch working? If so, can you please propose it for merging?

[Dan Streetman (ddstreet)]

Merge proposal created:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/lp1453330/+merge/319865

[Dan Streetman (ddstreet)]

Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

[Dmitry Shachnev (mitya57)]

> Dmitry, as you're following this bug and you have devel access to merge it, can you review/merge the change?

Sorry, I do not know the code much, so I would prefer someone else to review it. There are some other people with rights to merge subscribed here :)

[Dan Streetman (ddstreet)]

Iain, you created pull-lp-source, can you review this merge request?

[Iain Lane (laney)]

Ok, but you should look at the thing that I created and compare it to today's pull-lp-source. :)

[Khurshid Alam (khurshid-alam)]

I can confirm: the patch by Dan is woking!

[Michael Terry (mterry)]

The linked MP is marked as merged, but I don't think it is, looking at trunk. Dan, can you get your MP sorted and reviewable, and then subscribe sponsors again?

[Dan Streetman (ddstreet)]

Yep, I just changed it to work-in-progress, until i finish updating it so pull-debian-* works also.

[Dan Streetman (ddstreet)]

After quite a bit of major updating, I have some test pkgs built here:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330/+packages

it moves pull-lp-source to 'pull-pkg'; that is able to pull from debian, ubuntu, or uca, and it can pull source, binaries, ddebs, or di files. It can also simply list all a package's files.

There are small bash scripts to help calling pull-pkg:
pull-(lp|debian|uca)-(source|binaries|ddebs|di)

can anyone interested in this test out the updated scripts from the test package please? I'd like input before updating the merge request.

My changes are all in my git repo in the branch 'pull-lp':
https://code.launchpad.net/~ddstreet/+git/ubuntu-dev-tools/+ref/pull-lp

[Dan Streetman (ddstreet)]

Ok, I updated my merge request to a git-based merge request:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

also, I consolidated the script naming; so it's now:

pull-lp-source
pull-lp-debs
pull-lp-ddebs
pull-lp-udebs

which I think is more obvious, as to what you want to pull (instead of pull-lp-binaries and pull-lp-di).

And the renamed 'main' script is:

pull-pkg

laney, mterry, any chance you could review the patches for possible merging? I know it's quite a lot of code, but it does add a lot of useful functionality, I think.

[Dan Streetman (ddstreet)]

Also, I have pkgs built for xenial/yakkety/zesty with the latest merge proposal code, in this ppa:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330

[Dan Streetman (ddstreet)]

After quite a bit of patch editing and git rebasing of my working repo, I've re-opened the merge request. Any coredevs who are interested please review the patches if you have time:
https://code.launchpad.net/~ddstreet/ubuntu-dev-tools/+git/ubuntu-dev-tools/+merge/322863

[Dan Streetman (ddstreet)]

Sponsors, @mterry, @laney, with 17.10 release behind us, does anyone have time to review this merge request?

[Dan Streetman (ddstreet)]

@mapreri you've uploaded several of the most recent ubuntu-dev-tools changes, can you review this merge request?

[Mattia Rizzolo (mapreri)]

Most probably I can, yes. But it's not a quick thing, I need to find some appropriate time to review.

BTW, even after merging some thing I'd prefer if you could stick around and deal with upcoming bugs in pull-pkg in the next months.

[Dan Streetman (ddstreet)]

> But it's not a quick thing, I need to find some appropriate time to review.

absolutely, I'm hoping it can get merged by b release next year, but it's obviously not critical.

> BTW, even after merging some thing I'd prefer if you could stick around and deal with
> upcoming bugs in pull-pkg in the next months.

yes of course, I'm not going anywhere.

thanks!

[Dan Streetman (ddstreet)]

@mapreri, just checking if you have had any time for any review of this yet.