ddebs are not provided for cloud-archive packages

[Billy Olsen]

Binary packages included in the Ubuntu Cloud Archives need to also include debug builds (ddebs) in order to analyze crash files. These ddebs should also be archived for future analysis for the life of the support of the cloud-archive versions themselves.

This is currently done for Ubuntu Kernels and should also be done for the Ubuntu Cloud Archive packages, so there is a precedence here.

[Corey Bryant (corey.bryant)]

I've started discussion with slangasek to see if we can get ddebs for the cloud archive hosted on http://ddebs.ubuntu.com/. We'll likely not pick up on the discussion until after the new year. But we should be able to find a solution one way or the other.

[Dan Streetman (ddstreet)]

FYI related to this, I did some proposed updates to pull-lp-source to expand its common functionality, which led to adding a new pull-uca-ddebs interface that allows downloading any UCA package's ddebs (if ddebs are available in the UCA LP archive). That would help even if the uca ddebs aren't hosted at ddebs.ubuntu.com. Of course, if ddebs were not built with the UCA packages, it can't pull any, so it would help only with new UCA package builds that include ddebs.
https://bugs.launchpad.net/ubuntu/+source/ubuntu-dev-tools/+bug/1453330

I hope to get those changes merged into ubuntu-dev-tools for the Artful cycle.

[Corey Bryant (corey.bryant)]

We are now building and publishing debug symbols directly to the PPAs/archives for queens-staging, queens-proposed, and queens-updates. We'll look to enabling for prior UCAs in the not too distant future if all goes well with queens.

[Corey Bryant (corey.bryant)]

I didn't realize we already had ddebs enabled for Pike. Marked as fix-released for Pike.

[Corey Bryant (corey.bryant)]

Sorry, these are not fix-released yet. While the .ddebs are available in the cloud-archive PPAs, they are not yet published to the reprepro archive. So there's still some work to do here.

[Dan Streetman (ddstreet)]

> While the .ddebs are available in the cloud-archive PPAs

am I missing something? I don't see any ddebs for, e.g., qemu in pike:
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/pike-staging/+packages?field.name_filter=qemu&field.status_filter=published&field.series_filter=

I *do* see ddebs for queens, so maybe pike-staging was updated to build ddebs more recently than ^ pike qemu was built (13 hours ago)?

[Corey Bryant (corey.bryant)]

I wasn't considering staging. We really have 3 PPAs per UCA release: -staging, -proposed, and -updates. The first is just a place where we stage packages before promoting to proposed. The latter two you aren't public. Those are the real UCA that get synced to the real UCA via reprepro, from which you install after add-apt-repository cloud-archive:pike(-proposed).

[Dan Streetman (ddstreet)]

> We really have 3 PPAs per UCA release: -staging, -proposed, and -updates. The first is just
> a place where we stage packages before promoting to proposed. The latter two you aren't public.

Sad face.

Well that's a problem if you don't plan on including the ddebs in -staging, since that's the only public PPA. That's what the pull-uca-ddebs tool will be looking at, so if you aren't including ddebs into the (only) public UCA ppa, it won't find them.

Are you planning to enable ddeb building in the older -staging repos? It seems to be enabled in the queens-staging ppa already.

[Corey Bryant (corey.bryant)]

On Fri, Nov 17, 2017 at 1:43 PM, Dan Streetman <email address hidden>
wrote:

> > We really have 3 PPAs per UCA release: -staging, -proposed, and
> -updates. The first is just
> > a place where we stage packages before promoting to proposed. The latter
> two you aren't public.
>
> Sad face.
>
> Well that's a problem if you don't plan on including the ddebs in
> -staging, since that's the only public PPA. That's what the pull-uca-
> ddebs tool will be looking at, so if you aren't including ddebs into the
> (only) public UCA ppa, it won't find them.
>
> Are you planning to enable ddeb building in the older -staging repos?
> It seems to be enabled in the queens-staging ppa already.
>
>
>
We could but all packages get rebuilt once they hit proposed so the ddebs
may not match up anyway.

[Dan Streetman (ddstreet)]

> We could but all packages get rebuilt once they hit proposed so the ddebs
> may not match up anyway.

really? why are they rebuilt in the private ppa?

anyway, because pull-* uses the (public) lp api to find packages and enumerate the binary files they create, if the ddebs aren't listed in the public -staging ppa, it won't know about them even if they are included in the ubuntu-cloud.archive.canonical.com archive. So (assuming the ddeb filename isn't changing between -staging and -proposed) pull-uca-ddebs will first look on ubuntu-cloud for the ddeb files, and so should pull the correct (privately built) ones. So yeah, please do enable ddeb building in the public -staging ppas.

[Corey Bryant (corey.bryant)]

I think the main reason we rebuild in proposed is because we only build x86 in staging. So we need to rebuild in order to get binaries for other architectures. There may be other reasons.

I chatted with wolsen a bit about this and he helped me understand the advantage of pull-uca-ddebs vs apt installing ddebs. pull-uca-ddebs will pull ddebs from Launchpad, having access to all historical binaries because LP keeps them around. Whereas only the currently published ddebs would be available via apt installing from the UCA.

Other than PPA size, I don't see any reason why we couldn't build all arch's in staging and then just promote binaries without rebuild to -proposed. That would enable pull-uca-ddebs to pull from -staging, knowing that the corresponding versions in -proposed and -updates were not rebuilt.

I need to sync with James on this to figure out how best to move forward.

One question I do have is whether or not there is still any need to have ddebs published to the cloud archive if pull-uca-ddebs will be able to obtain a superset of what would be available via apt installing from the cloud-archive.

[Corey Bryant (corey.bryant)]

Just a note about copying without rebuild from staging->proposed in the context of PPA signing keys. Currently the staging PPA signing key differs from the private PPAs signing key. I believe that would prevent us from copying binaries (without rebuilding) from staging to proposed. If the keys were the same, then I believe copying would be an option. This may also be part of our security model for the cloud archive. I will discuss with James next week.

[Dan Streetman (ddstreet)]

> One question I do have is whether or not there is still any need to have ddebs published
> to the cloud archive if pull-uca-ddebs will be able to obtain a superset of what would
> be available via apt installing from the cloud-archive.

my MR for pull-uca-ddebs has been dragging on for a long, long time while I try to get someone to review and merge it into debian, so the public at large will have to be using apt to install uca ddebs (assuming anyone else needs the uca ddebs). And probably that will remain the common way to install ddebs for at least a while. However, as long as the ddebs are available in -staging, our team should be ok; I'll step back and let wolsen and you guys decide if the uca ddebs are needed in the apt archive as well as lp.

Also if you want to review and/or test pull-uca-ddebs (and associated scripts) I have a ppa I keep updated with my MR commits:
https://launchpad.net/~ddstreet/+archive/ubuntu/lp1453330

[Dan Streetman (ddstreet)]

note, i've just forked ubuntu-dev-tools since the debian maintainers decided my changes were too large and started ignoring all my communications ;-)

https://launchpad.net/~ddstreet/+archive/ubuntu/ubuntu-dev-tools

my fork includes pull-uca-debs and pull-uca-ddebs, but you need to of course actually build the ddebs and publish them in the ~ubuntu-cloud-archive ppas, for the tool to find them

[Dan Streetman (ddstreet)]

Also, is there any update on this? UCA still isn't building ddebs?

[Dan Streetman (ddstreet)]

To clarify:

the ddebs.ubuntu.com archive is, for the most part, not useful. It contains ddebs, but only for the last few versions. Many, possibly most, coredumps we see come from older packages. So we do not really care if you sync the UCA ddebs to ddebs.ubuntu.com or not, since it's unlikely we would actually bother to use it.

in launchpad, all ddebs built for ubuntu archive packages are stored for the life of the release, and are accessible either by a rather painful navigation by browser to get the specific ddeb for the right package/version, or simply by using pull-lp-ddebs. This is why we're asking you to make the UCA ddebs available in the UCA -staging ppa.

Related to this, it would really be nice if the -staging ppa binaries actually were the same as the published binaries in the UCA archive (instead of being rebuilt in a private ppa). Having the actual debs (and ddebs) in -staging (or if you can make -proposed public) would help to debug problems with the UCA packages.

To give you an idea of what we have to currently do to debug any coredump from any UCA package binary:

1) create a new PPA
2) edit the PPA 'dependencies' to depend on the corresponding UCA release -staging ppa
3) enable building dbgsym in that PPA
4) go to the UCA -staging ppa, e.g. mitaka-staging package details page
5) go to 'copy packages', and search for the specific package/version needed
6) select the right (older) version, use private ppa from step 1 as 'destination' ppa
7) make sure 'rebuild the copied sources' is checked, and then 'copy packages'

the pkg will rebuild in our private ppa, including dbgsyms. however, it's not an exact match, since all the associated packages that it builds with aren't exactly the same as the associated pkgs that the original pkg built with. And we have to do ^ for every UCA package we want dbgsyms for. Typically, the symbols are "close enough" to use to make sense of the core file, but this obviously is not great.

if you make the UCA ddebs (and actual real identical debs would be great too, not the not-really-the-same -staging debs) available in a public ppa (like -staging) then we can simply use pull-lp-ddebs to grab the specific ddebs we need, which will be both more accurate as well as vastly faster than our currently process to debug UCA packages.

[Corey Bryant (corey.bryant)]

On Thu, May 17, 2018 at 1:10 PM, Dan Streetman <email address hidden>
wrote:
<snip>

if you make the UCA ddebs (and actual real identical debs would be great
> too, not the not-really-the-same -staging debs) available in a public
> ppa (like -staging) then we can simply use pull-lp-ddebs to grab the
> specific ddebs we need, which will be both more accurate as well as
> vastly faster than our currently process to debug UCA packages.
>
>
I think we discussed this already but ddebs have been available in
queens-staging and will be available for rocky-staging (we only have python
packages in rocky-staging atm) and future -staging PPAs. Actually the same
is true for the -proposed/-updates private PPAs. The problem is that we
didn't find a way to get reprepro to publish the ddebs to
http://ubuntu-cloud.archive.canonical.com/ubuntu/.

Thanks,
Corey

[Corey Bryant (corey.bryant)]

I'm going to mark pre-Queens releases as Won't Fix. In discussions with wolsen we agreed to focus on ddebs only for Queens+.

[Corey Bryant (corey.bryant)]

Is this still considered Critical since we're now publishing ddebs to UCA PPAs?

[Dan Streetman (ddstreet)]

> ddebs have been available in queens-staging and will be available for rocky-staging

excellent thanks! I had not checked queens yet since we deal mostly with bugs from older releases.

> Actually the same is true for the -proposed/-updates private PPAs

is there any way for you to copy these backwards to -staging? or make -proposed and/or -updates public? or even (maybe temporarily) give us private access? if we can access them in LP, we don't need them to be published to the archive (though it's fine for you to eventually do that).

[Dan Streetman (ddstreet)]

> Is this still considered Critical since we're now publishing ddebs to UCA PPAs?

for queens and later, where we can get ddebs, no i think it's normal or low priority.

however for the older uca releases, since we still don't actually have access to the ddebs (even if they are privately available to you), I would say yes it's still critical to us.

[Corey Bryant (corey.bryant)]

I've gone ahead and enabled ddebs for older releases. However they will only get produced on new version backports. There are no plans to rebuild what's already been published. All of the following PPAs will now produce ddebs on new builds:

icehouse-staging/proposed/updates
mitaka-staging/proposed/updates
ocata-staging/proposed/updates
pike-staging/proposed/updates
queens-staging/proposed/updates
rocky-staging/proposed/updates

[Corey Bryant (corey.bryant)]

jamespage and I discussed making proposed/updates PPAs public in our team IRC channel this morning. We don't see any reason why they can't be public. They're currently under a private team on Launchpad. I'm going to look into creating a public team on Launchpad and migrating all existing binaries (no rebuilds) to the new public PPAS, and then updating the reprepro config to point at the public PPAs. This seems like it would make for a smooth transition, however there's always a chance that things don't go as planned once digging in. I will plan to work on this but it likely won't be for another week or two.

[Dan Streetman (ddstreet)]

just checking - any updates or changes for this?

[Corey Bryant (corey.bryant)]

Finally got this done for victoria, backing PPAs are now public. See: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/victoria-updates

For release prior to victoria, I don't plan to move the PPAs to public ones. Instead, I was thinking we can regularly binary packages that produce ddebs to similar public PPAs.

[Corey Bryant (corey.bryant)]

This is now fix released via the following PPAs for releases prior to Victoria. Please let us know if we are missing any packages. Note that early on we decided to enable ddeb generation for cloud archive PPAs, but only for newly built packages (ie. we wouldn't rebuild old packages to generate ddebs). Therefore some of the older package won't have ddebs, and newton was skipped entirely because it didn't have any package builds that generated ddebs. We will continue to promote new versions of the ddeb-related ppackages to these PPAs as packages get promoted to -updates.

sudo add-apt-repository ppa:ubuntu-cloud-archive/mitaka-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/ocata-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/pike-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/queens-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/rocky-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/stein-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/train-updates-ddeb
sudo add-apt-repository ppa:ubuntu-cloud-archive/ussuri-updates-ddeb