update-notifier should not alert users about esm-apps in a ESM machine

[Lucas Albuquerque Medeiros de Moura]

[Impact]
When users are running an ESM distro and looking at the MOTD they may be surprised by an alert generated by update-notifier that both esm-infra and esm-apps could be enabled in the machine to receive future updates. However, esm-apps will not bring much to an ESM distro. This means that users will be receiving alerts about esm-apps unnecessarily.

[Test case]

To reproduce the issue:

1. Launch a xenial container
2. Create the file /etc/ubuntu-advantage/uaclient.conf with the following content:

# Ubuntu-Advantage client config file.
contract_url: 'https://contracts.canonical.com'
security_url: 'https://ubuntu.com/security'
data_dir: /var/lib/ubuntu-advantage
log_level: debug
log_file: /var/log/ubuntu-advantage.log
features:
  allow_beta: true

3. Install ubuntu-advantage-client from this PPA:
https://code.launchpad.net/~ua-client/+archive/ubuntu/daily

PS: Make sure to not allow the installation to override the config file you have just created.

4. Update the version of update-notifier to the one in proposed
5. Run `/usr/lib/update-notifier/apt-check --human-readable`
6. Verify that both esm-infra and esm-apps alerts appear at the end of the message

To verify that the error is fixed:

1. Run the past scenario until step 3
2. Install the new update-notifier from this ppa:
https://launchpad.net/~lamoura/+archive/ubuntu/update-notifier-test-ppa/
3. Run `/usr/lib/update-notifier/apt-check --human-readable`
4. Verify that only esm-infra is now showing an alert message at the end

[Where problems could occur]

We are changing some aspects of the package that could generate problems:

1) We are changing the condition for when to output esm-apps alerts. If that condition is not right, we may miss displaying the messages to users that need them.
2) We are adding a new dependency to the package, lsb-release. But since the package was already relying on that dependency, we are just fixing a packaging mistake here.
3) Instead of relying on `lsb_release` for distro-info, we are now getting that info directly from `/etc/os-release`. Since is an attempt to not rely on subprocess python calls on that script. However, since we are adding more python code, logic errors can not happen because of that new chunk of code.
4) Since we are touching translatable message, translation errors can happen because of the modifications we are providing here.
5) We are now adding a ESM Apps headers for the case where users now have
esm-apps source files in the system. If our logic is incorrect, we could be displaying that info on situations that are not needed.

However, we believe that all of the risks mentioned above are manageable and should not block the changes we are bringing to the package.

[Discussion]
When running /usr/lib/update-notifier/apt-check on a ESM machine, we could alert the user about the existence of both esm-infra and esm-apps if the system has unauthenticated source files for both services. Therefore, we would display a message like this in the system:

--------------------------
Enable UA Apps: ESM to receive additional future security updates.
See https://ubuntu.com/security/esm or run: sudo ua status

Enable UA Infra: ESM to receive additional future security updates.
See https://ubuntu.com/security/esm or run: sudo ua status
----------------------------------

This is not ideal, since ESM machines will not gain much from esm-apps.
Also, this messaging may hide the importance of using esm-infra on ESM
machines, since the users will not know beforehand which service to
prioritize.

Furthermore, we are also bringing back the message:

0 updates can be applied immediately

When the system has no packages to install. This is because users found that omission confusing, since they thought that maybe something went wrong with motd if display no messages when the system is up to date.

Additionally, we are adding an esm-apps header to the `apt-check` message if the system is a LTS one with esm-apps source files on it. The header should behave exactly like the one we have for esm-infra:

 UA Apps: Extended Security Maintenance (ESM) is (enabled|disabled).

We doing this to be consistent with the message we have on ESM distros

Finally, we are no longer relying on `lsb_release` on `apt-check` to get distro info, like name and version. We are parsing `/etc/os-release` for that now. The rationale for this change is some errors we have saw regarding the use of `subprocess.check_output`. Since we can parse the os-release file directly, we believe it is safer to do so than to rely on shell out for getting that info.

== Changelog ==

  * debian/control: add lsb-release to the update-notifier-common depends as it is used by apt-cdrom-check. (LP: #1927996)
  * data/apt_check.py
    - Only show esm-apps alerts on lts and non-esm distros (LP: #1926990)
    - Update esm-infra alert for distros on ESM mode
    - Show message with number of upgradable packages even if that
      number is zero (LP: #1926819)
    - Get distro name and version directly from /etc/os-release
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.44

---------------
update-notifier (3.192.44) impish; urgency=medium

  [ Lucas Moura ]
  * data/apt_check.py
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

 -- Brian Murray <email address hidden> Thu, 13 May 2021 14:47:01 -0700

[Łukasz Zemczak (sil2100)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.40.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.192.40.3) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:21.04.10 (amd64, s390x, armhf, arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.192.30.8) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:20.04.10.7 (s390x, amd64, ppc64el, armhf, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.192.1.11) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:18.04.11.13 (i386, s390x, arm64, armhf, ppc64el, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lucas Albuquerque Medeiros de Moura (lamoura)]

I can confirm that the xenial, bionic, focal and hirsute packages are working as expected.

To perform the verification I have used the following script:

--------------------------------------
#!/bin/sh
set -x

series=xenial
name=$series-dev

function generate_motd_message() {
    message=$1
    echo "-----------------------------------------------------------"
    echo $message
    lxc exec $name -- /usr/lib/update-notifier/update-motd-updates-available --force
    lxc exec $name -- update-motd
    echo "-----------------------------------------------------------"
}

function turn_distro_into_esm_mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/is_esm_supported and is_not_currently_supported/True #comment/' /usr/lib/update-notifier/apt-check
}

function turn_distro_into_non_esm_mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/True #comment/False/' /usr/lib/update-notifier/apt-check
}

function setup_update_notifier() {
    lxc exec $name -- sh -c "cat <<EOF >/etc/apt/sources.list.d/ubuntu-$series-proposed.list
    deb http://archive.ubuntu.com/ubuntu/ $series-proposed restricted main multiverse universe"
    lxc exec $name -- apt update
    lxc exec $name -- sh -c "apt install update-notifier -yq > /dev/null"
}

function install_ua_from_branch() {
    lxc exec $name --cwd /var/tmp/uac -- apt-get update
    lxc exec $name --cwd /var/tmp/uac -- apt-get install make
    lxc exec $name -- git clone https://github.com/canonical/ubuntu-advantage-client.git /var/tmp/uac
    lxc exec $name --cwd /var/tmp/uac -- sh -c "make deps > /dev/null"
    lxc exec $name --cwd /var/tmp/uac -- sh -c "DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -us -uc > /dev/null"
    lxc exec $name -- apt-get remove ubuntu-advantage-tools --assume-yes
    lxc exec $name --cwd /var/tmp/uac -- dpkg -i /var/tmp/ubuntu-advantage-tools_27.0_amd64.deb
    lxc exec $name -- ua version
}

function install_ua() {
    lxc exec $name -- add-apt-repository ppa:ua-client/daily -y
    lxc exec $name -- sudo apt-get update
    lxc exec $name -- sudo apt-get install ubuntu-advantage-tools -y
    lxc exec $name -- ua version
    lxc exec $name -- sudo apt-get update
}

function ua_disable_esm_apps() {
    lxc exec $name -- sudo ua disable esm-apps
}

function install_all_upgrades() {
    lxc exec $name -- sudo sh -c "apt update > /dev/null"
    lxc exec $name -- sudo sh -c "apt upgrade -y > /dev/null"
}

function ua_attach() {
    lxc exec $name -- sed -i 's/contracts.can/contracts.staging.can/' /etc/ubuntu-advantage/uaclient.conf
    lxc exec $name -- ua attach $UACLIENT_BEHAVE_CONTRACT_TOKEN_STAGING
}

function ua_detach() {
    lxc exec $name -- ua detach --assume-yes
}

function generate_ua_motd_messages() {
    lxc exec $name -- python3 /usr/lib/ubuntu-advantage/ua_update_messaging.py
}

function install_update_motd() {
    lxc exec $name -- apt install update-motd -yq
}

function turn_esm_apps_into_non_beta() {
    lxc exec $name -- sh -c "echo 'features:\n allow_beta: true' >> /etc/ubuntu-advantage/uaclient.conf"
}

function update_contract_effectiveto() {
    operation=$1
    num_days=$2
    replace_d...

[Łukasz Zemczak (sil2100)]

Hey Lucas! What package versions did you use? Were those from the ubuntu archives? Since I see the instructions still mention using the PPA while for testing we need to use the archive packages.

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Hi Lukasz, those instructions were created before the package was uploaded into ṕroposed. In the script I have used to test this bug, I have used the proposed package.

But if needed, I can update the instructions to use the proposed packages instead

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.40.3

---------------
update-notifier (3.192.40.3) hirsute; urgency=medium

  * debian/control: add lsb-release to the update-notifier-common depends as
    it is used by apt-cdrom-check. (LP: #1927996)
  * data/apt_check.py
    - Only show esm-apps alerts on lts and non-esm distros (LP: #1926990)
    - Update esm-infra alert for distros on ESM mode
    - Show message with number of upgradable packages even if that
      number is zero (LP: #1926819)
    - Get distro name and version directly from /etc/os-release
    - Fallback to lsb_release command if there are any problems using
      /etc/os-release as source of information
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

 -- Lucas Moura <email address hidden> Fri, 14 May 2021 16:41:27 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.30.8

---------------
update-notifier (3.192.30.8) focal; urgency=medium

  * debian/control: add lsb-release to the update-notifier-common depends as
    it is used by apt-cdrom-check. (LP: #1927996)
  * data/apt_check.py
    - Only show esm-apps alerts on lts and non-esm distros (LP: #1926990)
    - Update esm-infra alert for distros on ESM mode
    - Show message with number of upgradable packages even if that
      number is zero (LP: #1926819)
    - Get distro name and version directly from /etc/os-release
    - Fallback to lsb_release command if there are any problems using
      /etc/os-release as source of information
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

 -- Lucas Moura <email address hidden> Fri, 14 May 2021 16:02:18 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.1.11

---------------
update-notifier (3.192.1.11) bionic; urgency=medium

  * debian/control: add lsb-release to the update-notifier-common depends as
    it is used by apt-cdrom-check. (LP: #1927996)
  * data/apt_check.py
    - Only show esm-apps alerts on lts and non-esm distros (LP: #1926990)
    - Update esm-infra alert for distros on ESM mode
    - Show message with number of upgradable packages even if that
      number is zero (LP: #1926819)
    - Get distro name and version directly from /etc/os-release
    - Fallback to lsb_release command if there are any problems using
      /etc/os-release as source of information
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

 -- Lucas Moura <email address hidden> Fri, 14 May 2021 15:43:17 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.168.15

---------------
update-notifier (3.168.15) xenial; urgency=medium

  * debian/control: add lsb-release to the update-notifier-common depends as
    it is used by apt-cdrom-check. (LP: #1927996)
  * data/apt_check.py
    - Only show esm-apps alerts on lts and non-esm distros (LP: #1926990)
    - Update esm-infra alert for distros on ESM mode
    - Show message with number of upgradable packages even if that
      number is zero (LP: #1926819)
    - Get distro name and version directly from /etc/os-release
    - Fallback to lsb_release command if there are any problems using
      /etc/os-release as source of information
    - Show esm-apps status header when running on LTS distro that
      has not yet entered Extended Security Maintenance

 -- Lucas Moura <email address hidden> Fri, 14 May 2021 15:00:04 -0300