ubuntu-pro-client-l10n translations are stripped

[Grant Orndorff]

[ Impact ]

In the upcoming version of ubuntu-advantage-tools, we will ship translation files in a new separate binary package called ubuntu-pro-client-l10n.

If you build ubuntu-pro-client-l10n with the current version of pkgbinarymangler, then translations are stripped and tarballed for inclusion in the langpack. We specifically don't want these translations included in the langpack for any release because pro-client gets updates (including strings) more frequently and many more times than langpacks get built.

In order to ship those translation files in ubuntu-pro-client-l10n we need to stop pkgbinarymangler from stripping them at build time. pkgbinarymangler has striptranslations.blacklist specifically for this purpose, so the fix is to add ubuntu-pro-client-l10n to that list.

[ Test Plan ]

Since the ubuntu-pro-client-l10n package is only getting introduced in the next version of ubuntu-advantage-tools (v30), the fixing and testing of this bug needs to be closely coordinated with the release of u-a-t.

Once u-a-t v30 is fully reviewed and ready to move to -proposed, we will first upload the fix of this bug to -proposed and wait for the binary proposed publication of pkgbinarymangler to complete. Then we will accept u-a-t v30 to -proposed.

In order to test that this bug is fixed, we will check that the binary ubuntu-pro-client-l10n package built from u-a-t v30 in -proposed contains the appropriate translation files. We will also verify that translations are working in u-a-t when ubuntu-pro-client-l10n is installed.

After showing all of that working together, we can mark this bug as verification-done

[ Where problems could occur ]

striptranslations.blacklist is a list of regexes, so if `ubuntu-pro-client-l10n` accidentally matches some other package, then that package would not have its translations stripped either.

Generally, new packages in -updates on ESM releases such as xenial and bionic (perhaps aside from u-a-t) are surprising and unexpected to users.

[Gunnar Hjalmarsson (gunnarhj)]

@Grant: There is a variable which disables pkgbinarymangler completely, and which can be set in debian/rules:

export NO_PKG_MANGLE=1

It's used by some other packages for this very purpose, and in this case it would mean that we could avoid all those pkgbinarymangler uploads.

Does pkgbinarymangler do anything important with ubuntu-advantage-tools (besides translation stripping)? If not, I think using the NO_PKG_MANGLE variable instead is worth considering.

[Grant Orndorff (orndorffgrant)]

Hi Gunnar, that's a great suggestion!

I've been using NO_PKG_MANGLE=1 for testing, but was worried about what else we'd miss if pkgbinarymangler was completely disabled. I believe it is at least stripping debug symbols from our apt-hook binary and also trimming our changelog on older releases.

But like you say, it would be nice to avoid all these uploads. I'll do some tests today to see exactly what the impact will be. If it is small enough then we can probably just use NO_PKG_MANGLE=1 and save us some trouble :)

[Grant Orndorff (orndorffgrant)]

I ran some tests and this is what I found:

I wasn't exactly right in thinking pkgbinarymangler is stripping debug symbols from our binary. It is dh_strip that is doing that, but on xenial, dh_strip honors the NO_PKG_MANGLE environment variable. However in this case the difference in resulting binary size is negligible (<1kb).

I found two things that pkgbinarymangler is doing for us:

1. it truncates the ubuntu-advantage-tools changelog to the latest 10 entries.
2. it links the ubuntu-advantage-pro changelog to the ubuntu-advantage-tools changelog.

So by setting NO_PKG_MANGLE=1 we end up with a u-a-t binary package that is 20kb larger (all from the changelog) and an ubuntu-advantage-pro binary package that is also 20kb larger (also all from the changelog since it is neither truncated nor linked). ubuntu-advantage-pro is only installed on Cloud Pro images, though, so is less important than ubuntu-advantage-tools which is everywhere.

It feels a bit wrong to ship a 20kb changelog when its not necessary, but maybe it's okay.

I think we should get an SRU team decision to decide which is preferable:

A. NO_PKG_MANGLE=1 and 20kb u-a-t changelog for x, b, f, j, l
B. SRU the striptranslations.blacklist update to x, b, f, j, l

[Robie Basak (racb)]

Speaking for myself as a member of the SRU team, and not the SRU team as a whole:

It seems to me that since pkgbinarymangler is designed to do multiple things, some of which we have no reason to disable, we should aim to disable just the translation handling since that's exactly what we want to do. And the package has a mechanism to do that so we should use it. Even if a little unwieldy by requiring a pkgbinarymangler upload, using this mechanism seems like it would be relatively risk-free in an SRU because it is very specifically targetted. Even if it does potentially impact every build, its behaviour should be able to be fairly easily checked. We could verify that its behaviour hasn't changed against an unrelated package during SRU verification easily and confidently enough. Using NO_PKG_MANGLE sounds like it has other side effects that might have more implications that need consideration and therefore more risk of missing something.

For the development release, perhaps a finer-grained control in pkgbinarymangler that can be operated from the build of a package itself rather than changing pkgbinarymangler would be appropriate. But for an SRU I think using striptranslations.blacklist seems like the minimally impactful way to do it.

[Grant Orndorff (orndorffgrant)]

Thanks Robie!

We'll stick with the plan to update striptranslations.blacklist via pkgbinarymangler SRUs then.

Since this is tightly coupled to the upcoming u-a-t release, we'll handle the uploads as part of the u-a-t release process.

[Andreas Hasenack (ahasenack)]

This is in mantic already: https://launchpad.net/ubuntu/+source/pkgbinarymangler/154

[Andreas Hasenack (ahasenack)]

Hello Grant, or anyone else affected,

Accepted pkgbinarymangler into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/152.23.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Grant, or anyone else affected,

Accepted pkgbinarymangler into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/149.22.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Grant, or anyone else affected,

Accepted pkgbinarymangler into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/144.20.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Grant, or anyone else affected,

Accepted pkgbinarymangler into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/138.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Grant, or anyone else affected,

Accepted pkgbinarymangler into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pkgbinarymangler/129.16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Renan Rodrigo (renanrodrigo)]

As part of the validation process for https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2038461, as mentioned in the test plan, we ran tests which depended on pkgbinarymangler proposed version to run. Those are the i18n.feature tests, which failed before, as the translations were stripped. The tests are passing now.

The ubuntu-advantage-tools package in -proposed has been built with this -proposed version of pkgbinarymangler (checked amd64 for simplicity):

ubuntu-advantage-tools 30~16.04 was built with
pkgbinarymangler 129.16.04.0
(from build log https://launchpadlibrarian.net/697068643/buildlog_ubuntu-xenial-amd64.ubuntu-advantage-tools_30~16.04_BUILDING.txt.gz)

ubuntu-advantage-tools 30~18.04 was built with
pkgbinarymangler 138.18.04.2
(from build log https://launchpadlibrarian.net/697068270/buildlog_ubuntu-bionic-amd64.ubuntu-advantage-tools_30~18.04_BUILDING.txt.gz)

ubuntu-advantage-tools 30~20.04 was built with
pkgbinarymangler 144.20.04.0
(from build log https://launchpadlibrarian.net/697068151/buildlog_ubuntu-focal-amd64.ubuntu-advantage-tools_30~20.04_BUILDING.txt.gz)

ubuntu-advantage-tools 30~22.04 was built with
pkgbinarymangler 149.22.04.0
(from build log https://launchpadlibrarian.net/697068014/buildlog_ubuntu-jammy-amd64.ubuntu-advantage-tools_30~22.04_BUILDING.txt.gz)

ubuntu-advantage-tools 30~23.04 was built with
pkgbinarymangler 152.23.04.0
(from build log https://launchpadlibrarian.net/697067775/buildlog_ubuntu-lunar-amd64.ubuntu-advantage-tools_30~23.04_BUILDING.txt.gz)

- All the versions above are the -proposed version of the packages
- I used amd64 as the example but all build logs confirm those versions

I am thus marking this bug as verification-done.

[Renan Rodrigo (renanrodrigo)]

As the CI coverage for the u-a-t SRU only checks for the translations to be working on B/F/J, I am adding new runs for L/M here, and a manual run for X (xenial has a bug which prevents our automated tests from checking translations, but it is working)

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 152.23.04.0

---------------
pkgbinarymangler (152.23.04.0) lunar; urgency=medium

  * striptranslations.blacklist: Add ubuntu-pro-client-l10n. (LP: #2037584)

 -- Grant Orndorff <email address hidden> Wed, 27 Sep 2023 11:49:44 -0400

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for pkgbinarymangler has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 149.22.04.0

---------------
pkgbinarymangler (149.22.04.0) jammy; urgency=medium

  * striptranslations.blacklist: Add ubuntu-pro-client-l10n. (LP: #2037584)

 -- Grant Orndorff <email address hidden> Wed, 27 Sep 2023 11:49:44 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 144.20.04.0

---------------
pkgbinarymangler (144.20.04.0) focal; urgency=medium

  * striptranslations.blacklist: Add ubuntu-pro-client-l10n. (LP: #2037584)

 -- Grant Orndorff <email address hidden> Wed, 27 Sep 2023 11:49:44 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 138.18.04.2

---------------
pkgbinarymangler (138.18.04.2) bionic; urgency=medium

  * striptranslations.blacklist: Add ubuntu-pro-client-l10n. (LP: #2037584)

 -- Grant Orndorff <email address hidden> Wed, 27 Sep 2023 11:49:44 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package pkgbinarymangler - 129.16.04.0

---------------
pkgbinarymangler (129.16.04.0) xenial; urgency=medium

  * striptranslations.blacklist: Add ubuntu-pro-client-l10n. (LP: #2037584)

 -- Grant Orndorff <email address hidden> Wed, 27 Sep 2023 11:49:44 -0400