pkg-create-dbgsym creates -dbgsym packages with the source version, not the binary version

[Matthias Klose]

[SRU Justification]
seen in
https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test/+sourcepub/4962729/+listing-archive-extra
https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test/+sourcepub/4962755/+listing-archive-extra

The gcc-5 (and gcc-4-x) source packages build some binary packages where the binary version doesn't match the source version, but the -dbgsym package is still built using the source version of the package.

These packages now all fail to upload, in every release ...

[Test case]
1. Turn on the ddeb flag in launchpad
2. Upload gcc to the ubuntu-toolchain-r ppa
3. Verify that the package does not fail to upload

[Regression Potential]
This can't break user systems, just building of future updates. Known-weird packages like gcc, binutils, systemd, linux and others have been tested against the new version and build correctly, giving a strong indication that everything is good.

There will probably be other failed to upload cases from enabling native ddebs, but they were always going to happen with or without this fix. In any failure case, we can unbreak the archive or push through an urgent update by building it in a PPA that doesn't have the ddebs flag enabled, or that has a reverted pkg-create-dbgsym.

[Colin Watson (cjwatson)]

18:56 <cjwatson> if I were fixing this pkg-create-dbgsym thing, I believe I would split it into two pieces, and run the second half from dh_gencontrol
18:57 <cjwatson> that way it can have the -v argument handy

[Matthias Klose (doko)]

<wgrant> ddebs flag disabled everywhere, infinity rebuilding the only bad builds targeted to the primary archive (a security update for four releases)

[Martin Pitt (pitti)]

I have this working in the packaging branch.

There already was a disabled test case "dhtest.versionoverride" for a package that spits out a different binary version. This commit enables this now, and adds an additional check that the binary versions of deb and ddeb are identical:
  http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/pkg-create-dbgsym/vivid/revision/234

I also added an additional test for a package that calls pkg_create_dbgsym directly and does not use debhelper, like e. g. binutils does:
  http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/pkg-create-dbgsym/vivid/revision/235

I rearchitectured the ddeb building, so that with debhelper dh_strip now only prepares the debian/pkgname-dbgsym/ package build dir, and diverts dh_gencontrol which adjusts the Version: and Depends: and then builds the .ddeb. This looks like a lot of code, but most of it (the whole argument parsing, determination of which binary packages to process, and building of the .ddebs) is a straight copy&paste of the already existing dh_strip. The only "new" code here is

    # adjust ddeb version from deb, for dpkg-gencontrol's -v option
    debversion=`grep '^Version:' $pkgdir/DEBIAN/control | cut -f2- -d\ `
    sed -i "s/^Version: .*\$/Version: $debversion/; /^Depends:/ s/(= .*)/(= $debversion)/" $ddebpkgdir/DEBIAN/control

which addresses this bug, i. e. adjusts the ddeb versions for dpkg-gencontrol -v.

The existing test cases still all succeed, and the test suite runs during package build, so if the SRUed package builds we can already be fairly confident that there is no regression. Nevertheless, verifying that gcc now builds in a PPA with ddebs enabled should be done to verify the fix, and this test rebuild should also include at least systemd (many binaries with associated ddebs, debhelper) and binutils (manual pkg_create_dbgsym, no debhelper) to ensure no regressions.

But for the record, and "SRU regression potential": Nevertheless there is always the potential to break a package which previously built, of course. Or that we suddenly stop producing ddebs for a package with a weird debian/rules (e. g. using dh_strip but not dh_gencontrol).

[Martin Pitt (pitti)]

The new mangler is in https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test/+packages , together with source copies of binutils, systemd, and gcc-4.9.

systemd built fine, and e. g. https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test/+build/7362655 has the expected -dbgsym packages.

binutils built fine (https://launchpadlibrarian.net/204624531/buildlog_ubuntu-vivid-amd64.binutils_2.25-5ubuntu7_BUILDING.txt.gz) and the build log shows that it produced the -dbgsym, but as the package calls pkg_create_dbgsym directly and without -a, they don't end up in the .changes file and thus aren't exported. We need to fix that once we enable the native ddeb support in launchpad.

gcc-4.9 is still building.

[Martin Pitt (pitti)]

gcc-4.9 finished building. The ddebs now seem to have correct versions:

 Package: fixincludes-dbgsym
 Source: gcc-4.9
 Version: 1:4.9.2-10ubuntu13

(i. e. include the epoch) Nevertheless it still failed to upload with https://launchpadlibrarian.net/204975438/upload_7362665_log.txt:

INFO Upload was rejected:
INFO Unable to find source publication gcc-4.9/1:4.9.2-10ubuntu13 in vivid

What does that mean?

[Martin Pitt (pitti)]

Ah, it needs to look like that:

  Source: gcc-4.9 (4.9.2-10ubuntu13)

This needs fixing in pkg-create-dbgsym.

[Martin Pitt (pitti)]

Source: versioning fixed in http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/pkg-create-dbgsym/vivid/revision/240 (with tests). Reuploaded as 0.65pitti2 to the PPA.

I also uploaded gcc-4.9 with disabling the tests (uncommented "with_check := disabled by hand" in debian/rules.defs), so that this builds a lot faster.

I uploaded the same version to utopic and trusty; utopic and vivid have the same version, and trusty is missing this one:

  http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/pkg-create-dbgsym/vivid/revision/232

IMHO this is a fix which we should have in trusty as well, for correct Architecture: fields.

Precise has a much older version, I'll backport the fixes there instead. As for lucid, as that will be EOL in about two days I figure it doesn't make much sense any more to backport there?

[Martin Pitt (pitti)]

I backported the fixes and tests for the Source:/Version: as well as Colin's Architecture: handling to precise. It is in the PPA now.

[Martin Pitt (pitti)]

gcc built on all releases now, and spot-checking ddebs looks fine. E. g.

   https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test/+build/7366154/+files/fixincludes-dbgsym_4.9.1-16ubuntu6pitti1_amd64.ddeb

has

 Package: fixincludes-dbgsym
 Source: gcc-4.9 (4.9.1-16ubuntu6pitti1)
 Version: 1:4.9.1-16ubuntu6pitti1
 Depends: libgcc1-dbg (= 1:4.9.1-16ubuntu6pitti1)

and uploading went alright. The package is supposed to be empty as the debug symbols are already in libgcc1-dbg, which it depends on.

The precise build https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test/+build/7366150/+files/fixincludes-dbgsym_4.6.3-1ubuntu5pitti1_amd64.ddeb has

 Package: fixincludes-dbgsym
 Source: gcc-4.6 (4.6.3-1ubuntu5pitti1)
 Version: 1:4.6.3-1ubuntu5pitti1
 Depends: fixincludes (= 1:4.6.3-1ubuntu5pitti1)

which also is correct (epoch'ed, correct Source: version, upload went fine), and has debug symbols as in precise gcc-4.6 didn't yet build -dbg packages on its own.

binutils and systemd also built everywhere, e. g. systemd on trusty has an epoch'ed gudev lib: https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test/+files/libgudev-1.0-0-dbgsym_204-5ubuntu20_amd64.ddeb

 Package: libgudev-1.0-0-dbgsym
 Source: systemd (204-5ubuntu20)
 Version: 1:204-5ubuntu20
 Depends: libgudev-1.0-0 (= 1:204-5ubuntu20)

and contains debug symbols.

I am now reasonably confident with these fixes, and uploaded them to {precise,trusty,utopic,vivid}-proposed SRU review queues.

[Steve Langasek (vorlon)]

Hello Matthias, or anyone else affected,

Accepted pkg-create-dbgsym into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkg-create-dbgsym/0.66~trusty in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Matthias, or anyone else affected,

Accepted pkg-create-dbgsym into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkg-create-dbgsym/0.50.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Matthias, or anyone else affected,

Accepted pkg-create-dbgsym into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkg-create-dbgsym/0.66~utopic in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Hello Matthias, or anyone else affected,

Accepted pkg-create-dbgsym into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkg-create-dbgsym/0.66 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Pitt (pitti)]

I reconfigured https://launchpad.net/~ubuntu-core-dev/+archive/ubuntu/ddeb-test to use packages from -proposed, and removed the pkg-create-dbgsym packages from it. So we can re-use this PPA for verifying the proposed packages.

[William Grant (wgrant)]

I've built systemd, gcc and binutils for all architectures in https://launchpad.net/~wgrant/+archive/ubuntu/p-c-d-test/+packages, and everything works. Looks good to me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkg-create-dbgsym - 0.50.1

---------------
pkg-create-dbgsym (0.50.1) precise-proposed; urgency=medium

  [ Colin Watson ]
  * Check each element of Architecture separately using dpkg-architecture,
    rather than inaccurately checking the whole field for "any" or
    "linux-any".

  [ Martin Pitt ]
  * testsuite: Enable dhtest.versionoverride, and verify that ddeb has exact
    same version as the corresponding deb. Reproduces LP #1448247.
  * testsuite: Add "manualtest" which builds a package without debhelper or
    any other abstractions, and calls pkg_create_dbgsym directly.
  * pkg_create_dbgsym: Add -B option for only building the ddeb package dir in
    debian/ but not building the .ddeb package yet. This keeps the default
    behaviour for packages which call this manually, but allows us to modify
    the ddeb package directory later during the package build process.
  * Handle binary packages with different version than the source:
    - Add our own dh_gencontrol wrapper which adjusts the ddeb's versions
      after the original dh_gencontrol determined the final binary version of
      the original debs, and then builds the .ddebs.
    - dh_strip: Call pkg_create_dbgsym with -B, so that we move the actual
      .ddeb generation into dh_gencontrol.
    - preinst, postrm: Divert dh_gencontrol.
    - LP: #1448247

 -- Martin Pitt <email address hidden> Tue, 28 Apr 2015 16:08:28 +0200

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for pkg-create-dbgsym has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkg-create-dbgsym - 0.66~trusty

---------------
pkg-create-dbgsym (0.66~trusty) trusty-proposed; urgency=medium

  * Backport to trusty.

pkg-create-dbgsym (0.66) vivid; urgency=medium

  * testsuite: Enable dhtest.versionoverride, and verify that ddeb has exact
    same version as the corresponding deb. Reproduces LP #1448247.
  * testsuite: Add "manualtest" which builds a package without debhelper or
    any other abstractions, and calls pkg_create_dbgsym directly.
  * pkg_create_dbgsym: Add -B option for only building the ddeb package dir in
    debian/ but not building the .ddeb package yet. This keeps the default
    behaviour for packages which call this manually, but allows us to modify
    the ddeb package directory later during the package build process.
  * Handle binary packages with different version than the source:
    - Add our own dh_gencontrol wrapper which adjusts the ddeb's versions
      after the original dh_gencontrol determined the final binary version of
      the original debs, and then builds the .ddebs.
    - dh_strip: Call pkg_create_dbgsym with -B, so that we move the actual
      .ddeb generation into dh_gencontrol.
    - preinst, postrm: Divert dh_gencontrol.
    - LP: #1448247
  * debian/control: Drop obsolete (pre-lucid) Build-Conflicts.
  * Bump Standards-Version to 3.9.6 (no changes required).

pkg-create-dbgsym (0.65) utopic; urgency=medium

  * Check each element of Architecture separately using dpkg-architecture,
    rather than inaccurately checking the whole field for "any" or
    "linux-any".

 -- Martin Pitt <email address hidden> Tue, 28 Apr 2015 16:06:45 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkg-create-dbgsym - 0.66~utopic

---------------
pkg-create-dbgsym (0.66~utopic) utopic-proposed; urgency=medium

  * Backport to utopic.

pkg-create-dbgsym (0.66) vivid; urgency=medium

  * testsuite: Enable dhtest.versionoverride, and verify that ddeb has exact
    same version as the corresponding deb. Reproduces LP #1448247.
  * testsuite: Add "manualtest" which builds a package without debhelper or
    any other abstractions, and calls pkg_create_dbgsym directly.
  * pkg_create_dbgsym: Add -B option for only building the ddeb package dir in
    debian/ but not building the .ddeb package yet. This keeps the default
    behaviour for packages which call this manually, but allows us to modify
    the ddeb package directory later during the package build process.
  * Handle binary packages with different version than the source:
    - Add our own dh_gencontrol wrapper which adjusts the ddeb's versions
      after the original dh_gencontrol determined the final binary version of
      the original debs, and then builds the .ddebs.
    - dh_strip: Call pkg_create_dbgsym with -B, so that we move the actual
      .ddeb generation into dh_gencontrol.
    - preinst, postrm: Divert dh_gencontrol.
    - LP: #1448247
  * debian/control: Drop obsolete (pre-lucid) Build-Conflicts.
  * Bump Standards-Version to 3.9.6 (no changes required).

 -- Martin Pitt <email address hidden> Tue, 28 Apr 2015 16:06:45 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkg-create-dbgsym - 0.66

---------------
pkg-create-dbgsym (0.66) vivid; urgency=medium

  * testsuite: Enable dhtest.versionoverride, and verify that ddeb has exact
    same version as the corresponding deb. Reproduces LP #1448247.
  * testsuite: Add "manualtest" which builds a package without debhelper or
    any other abstractions, and calls pkg_create_dbgsym directly.
  * pkg_create_dbgsym: Add -B option for only building the ddeb package dir in
    debian/ but not building the .ddeb package yet. This keeps the default
    behaviour for packages which call this manually, but allows us to modify
    the ddeb package directory later during the package build process.
  * Handle binary packages with different version than the source:
    - Add our own dh_gencontrol wrapper which adjusts the ddeb's versions
      after the original dh_gencontrol determined the final binary version of
      the original debs, and then builds the .ddebs.
    - dh_strip: Call pkg_create_dbgsym with -B, so that we move the actual
      .ddeb generation into dh_gencontrol.
    - preinst, postrm: Divert dh_gencontrol.
    - LP: #1448247
  * debian/control: Drop obsolete (pre-lucid) Build-Conflicts.
  * Bump Standards-Version to 3.9.6 (no changes required).

 -- Martin Pitt <email address hidden> Tue, 28 Apr 2015 16:05:41 +0200

[Martin Pitt (pitti)]

I copied all of them to *-security as well, for the record.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkg-create-dbgsym - 0.67

---------------
pkg-create-dbgsym (0.67) vivid; urgency=medium

  * testsuite: Add dhtest.customdbgsym for a package that produces its own
    -dbgsym package, like our "linux" source package. Reproduces LP #1450464.
  * pkg_create_dbgsym: When called with -B (i. e. from dh_strip), create a
    marker for dh_gencontrol, so that the latter does not try to process
    custom -dbgsym binaries. (LP: #1450464)
  * dh_strip: Don't try to process *-dbgsym binaries.

 -- Martin Pitt <email address hidden> Thu, 30 Apr 2015 16:12:05 +0200