apt-check needs to consider the pro client's esm cache for updates when esm is disabled

[Renan Rodrigo]

[ Impact ]

Currently, the Pro Client sets up ESM sources in the system's APT configuration to advertise ESM updates even when the service is disabled. This is undesired, as described in

https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1990378

The next version of ubuntu-advantage-client will not bring this configuration anymore, and will remove it from systems where it exists.

Update Notifier relies on those unauthenticated ESM sources. It needs instead to rely on the Pro Client ESM apt cache to provide this information.

If this is not changes, users will not see any ESM update or advertisement if they don't have the services enabled.

The change in the reporting strategy will land in ubuntu-advantage-tools and will be SRUed back to Xenial, Bionic, Focal and Jammy.

[ Test Plan ]

In a Ubuntu LTS system (Xenial for esm-infra, bionic/focal for esm-apps):
  - Verify that the ubuntu-advantage-tools version installed is greater than 27.12
  - Verify that no ESM updates are reported when running /usr/lib/update-notifier/apt-check --human-readable
  - Install a version of update-notifier with the change (from -proposed, or from the MR branch if not uploaded yet)
  - Verify that ESM updates are when running /usr/lib/update-notifier/apt-check --human-readable

[ Where problems could occur ]

In the event of the change being implemented in a wrong/incomplete way, the side-effect is that users would not see the ESM updates, same as not implementing the change. However, it is simple enough to verify if this happens, and it is highly improbable that it does.

[ Other Info ]

Instead of reading the esm apt cache from the Pro Client, update-notifier could query the 'packages' API provided in the Client itself. However, for that to happen, it would make sense for the package to depend (or at least recommend) ubuntu-advantage-tools.

Even using the esm apt cache would be a fair reason for update-notifier to recommend ubuntu-advantage-tools - but it is the same kind of coupling (relying on implementation) as reading the "never" pin from esm repositories, so there is no hard requirement to add the soft dependency.

There is no upload to Kinetic because there are no ESM packages there. There is no report of ESM data today, and there will be no report even if this lands there.

For future changes, we want to redesign how this interaction between ubuntu-advantage-tools and update-notificer works, and then the definitive solution should land on all supported releases.

[Renan Rodrigo (renanrodrigo)]

On a separate note: this also affects Xenial, and needs to be SRUed to Xenial - however no Xenial branch was found to send the patch to. What would be the steps to do so?

[Renan Rodrigo (renanrodrigo)]

To anyone the core-dev team reading this:
the Pro Client has recently implemented exactly this feature: providing upgrade counts for installed packages. If you want, instead of bringing it from the cache as done in the MRs for this bug, we can change it to read the Pro Client data, as mentioned in the [Other info] section above.

Implementation would be more reliable and clean.

IMHO this would make update-notifier depend on u-a-t, but I don't think this is a big problem, given u-a-t is in minimal anyway.

Please let me know if you think it is ok to import and run the code from `uaclient`, and if it is fine to depend on the package for the counts.

[Julian Andres Klode (juliank)]

I'd rather we have the implementation in update-manager reused by pro and not the other way around, the whole thing should be meaningful for users.

Notably apt has its own implementation of phased updates and phasing that update-manager and update -notifier also needs to switch to, but which is not yet exposed in python-apt.

We can't have multiple implementations of phased updates that each count slightly different because their languages have different RNG implementations.

[Renan Rodrigo (renanrodrigo)]

> I'd rather we have the implementation in update-manager reused by pro and not the other way around, the whole thing should be meaningful for users.

I see the point, but what I took into consideration for this particular change is:

- Right now, update-notifier already depends on u-a-t implementation (not only code, but the details, like the soon-to-be-gone 'never' pinning on esm when disabled) to count and show esm updates when the services are disabled. With this change, we can make it keep depending on implementation, or depend on functionality directly, which is already implemented in u-a-t.

- The change sent in the MR keeps all logic as-is for update-notifier, including the dependency on the Pro Client for only those two things: checking if esm is enabled or not, and counting esm upgrades when the services are disabled.

- Even if u-a-t is to consume from update-notifier any information about standard/security upgrades, it should still take care of esm, as we discussed many times up to this point - Pro related things should live in the Pro client. Take the esm cache itself: it would not make much sense to create and maintain it under update-notifier, u-a-t should take care of that. So, I understand and agree that a single implementation is better and if we put esm upgrade counts to u-a-t and the other upgrade counts in update-notifier, we'll end up with both tools depending on each other to show complete and meaningful information; but it is not clear (at least to me, sorry, maybe I miss some information) that update-notifier should take this responsibility.

- I don ´t really understand how we can't treat phased updates in u-a-t for instance (which today we do not, btw, by design) - update manager has a python implementation and that's what update-notifier is relying on anyway. If/when the apt phasing implementation gets to python-apt, it will be available for u-a-t as well. But imho this is a separate issue and can be solved in a separate thread.

- Last, let's say we agree that the way u-a-t should consume information from update-notifier and/or vice-versa need to be reworked, this would require a redesign of how this should work before actually implementing something, and (you would not believe that) we have a deadline from the products side - u-a-t needs to SRU changes by January 26th, and it will impact update-notifier anyway. So, what I ask is: can we proceed with updating update-notifier now adapting to the changes in the client? and then (for next cycle, why not), I will be most happy to work on a better design and implementation of how package upgrades can be advertised.

[Julian Andres Klode (juliank)]

I specifically replied to #2 regarding possible future changes, not this change itself.

[Renan Rodrigo (renanrodrigo)]

I see. Sorry for being overdefensive in the answer.

My point in #2 is that, in this changeset itself, we could be importing uaclient and using functionality instead of relying on the implementation detail, before we think about a redesign.

Either way, please let me know what is the best approach and how to better sort this.

[Julian Andres Klode (juliank)]

So I thought I made it clear in the previous message that this wasn't blocked on me, but apparently not, and nobody reviewed the merges it turns out, so I went ahead and approved them. Minimal changes make for good SRU.

Let's figure the rest out when there's no time pressure.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.61

---------------
update-notifier (3.192.61) lunar; urgency=medium

  [ Renan Rodrigo Barbosa]
  * Rely on the Pro Client apt esm cache to check for esm updates
    (LP: #2002168)

 -- Christian Ehrhardt <email address hidden> Thu, 19 Jan 2023 14:03:22 +0100

[Andreas Hasenack (ahasenack)]

The xenial upload in unapproved does not have the fix for this bug here, was that intentional? It just has the fix for the translation crash:

https://launchpadlibrarian.net/647743365/update-notifier_3.168.19_source.changes

Launchpad-Bugs-Fixed: 2003543
Changes:
 update-notifier (3.168.19) xenial; urgency=medium
 .
   * po/*.po: do not translate template variable needed for
     package-data-downloader script (LP: #2003543)

[Andreas Hasenack (ahasenack)]

Hello Renan, or anyone else affected,

Accepted update-notifier into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.54.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Renan, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.192.30.16) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

update-notifier/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

@Andeas
The xenial upload in unapproved does not have the fix for this bug here, was that intentional? It just has the fix for the translation crash.

No that is wrong - I had this in unapproved:
 update-notifier (3.168.19) xenial; urgency=medium
 .
   [ Renan Rodrigo Barbosa]
   * Rely on the Pro Client apt esm cache to check for esm updates
     (LP: #2002168)
=> On Jan 19 15:36

Then it was replaced by the current one you linked to when Sebastien sponsored the translation fix.
I have provided a new (still 3.168.19 as it was not yet accepted) into xenial-unapproved.

[Christian Ehrhardt  (paelzer)]

@Andreas
For Bionic - Now also a https://launchpadlibrarian.net/647902226/update-notifier_3.192.1.17_source.changes showed up in -unapproved which needs a similar fix on top (the regen of the autogen files was incomplete). On this upload, since tagging is used I need to bump the version.

Coordinating with Lucas to add the missing bits I've uploaded (and made available in git) => 3.192.1.18
=> https://launchpadlibrarian.net/647903734/update-notifier_3.192.1.18_source.changes

[Andreas Hasenack (ahasenack)]

Hello Renan, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Renan, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Renan Rodrigo (renanrodrigo)]

I have performed the verification for this bug's fix using the package from proposed.
I just uploaded the logs with all commands executed as the test steps suggests (Yeah I know, I could have a tarball... my bad.)

I am marking verification-done for the LTS series.

[Andreas Hasenack (ahasenack)]

I verified the attached logs and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions.

There is no SRU freeze ongoing at the moment.

There is a halted phasing on the previous update due to #2003543, for which this upload includes a fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.54.5

---------------
update-notifier (3.192.54.5) jammy; urgency=medium

  * po/*.po: do not translate template variable needed for
    package-data-downloader script (LP: #2003543)

update-notifier (3.192.54.4) jammy; urgency=medium

  [ Renan Rodrigo Barbosa]
  * Rely on the Pro Client apt esm cache to check for esm updates
    (LP: #2002168)

 -- Lucas Moura <email address hidden> Fri, 20 Jan 2023 15:50:39 -0300

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.30.16

---------------
update-notifier (3.192.30.16) focal; urgency=medium

  * po/*.po: do not translate template variable needed for
    package-data-downloader script (LP: #2003543)

update-notifier (3.192.30.15) focal; urgency=medium

  [ Renan Rodrigo Barbosa]
  * Rely on the Pro Client apt esm cache to check for esm updates
    (LP: #2002168)

 -- Lucas Moura <email address hidden> Fri, 20 Jan 2023 16:17:53 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.1.18

---------------
update-notifier (3.192.1.18) bionic; urgency=medium

  * po/*.po: do no translate template variable needed for
    package-data-downloader script (LP: #2003543)
  * Bring generated autoconf files from the archive package (LP: #1999897)

update-notifier (3.192.1.16) bionic; urgency=medium

  [ Renan Rodrigo Barbosa]
  * Rely on the Pro Client apt esm cache to check for esm updates
    (LP: #2002168)

 -- Lucas Moura <email address hidden> Fri, 20 Jan 2023 16:25:17 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.168.19

---------------
update-notifier (3.168.19) xenial; urgency=medium

  [ Renan Rodrigo Barbosa]
  * Rely on the Pro Client apt esm cache to check for esm updates
    (LP: #2002168)

  [ Lucas Moura ]
  * po/*.po: do not translate template variable needed for
    package-data-downloader script (LP: #2003543)

 -- Christian Ehrhardt <email address hidden> Thu, 19 Jan 2023 15:21:16 +0100