update-notifier does not handle ESM Apps in the apt-check script

[Lucas Albuquerque Medeiros de Moura]

[Impact]
When ESM Apps goes into production we want to be able to advertise it to our users through MOTD.
Currently, update-notifier apt-check does not take into consideration the exclusive packages of ESM Apps, which is a problem.

[Test case]

To reproduce the issue, you can:

1. Launch a xenial vm
2. Install the latest version of uaclient from the stable ppa:
   https://launchpad.net/~ua-client/+archive/ubuntu/stable/
3. Change the following line on `/etc/ubuntu-advantage/uaclient.conf` from

   contract_url: 'https://contracts.canonical.com'

   To:

   contract_url: 'https://contracts.staging.canonical.com'

4. Attach a staging token to uaclient thorough:

    ua attach YOUR_STAGING_TOKEN

5. Run /usr/lib/update-notifier/apt-check --human-readable
6. Verify that no mention to ESM Apps is made and that update-notifier puts all of the ESM packages in the same bucket

To verify that the error is fixed:

1.Perform all the stages above until step 4
2 Install the new update-notifier from this ppa:
  https://launchpad.net/~lamoura/+archive/ubuntu/update-notifier-test-ppa
3. Verify now that ESM Apps is reflected in the message

[Where problems could occur]

The changes in this package should only be seen when MOTD is getting a new message. If that script fails for some reason, it seems that MOTD will only not present the message, which is doesn't seem to be a system critical issue.

[Discussion]

With ESM Apps going to production soon, we have decided to update the messages delivered by update-notifier apt-check to address the package count of ESM Apps and the possibility of installing more upgrades if the user has ESM Apps disabled.

We are also updating other parts of the messaging as well. First, we only display ESM Infra status
on ESM distros. However, we will keep showing the ESM Infra package count on all of them.

For ESM Apps, we are only performing the alerts (For example, that you might have x packages updates if ESM Apps is installed) if the user is on a LTS distro.

Finally, we are also addressing this bug here:
https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1883315

Where we were incorrectly counting ESM packages even if the user did not have access to those services.

== Changelog ==

  + data/apt-check
    - Make distinction of ESM Apps and ESM Infra when doing package count
    - Only count ESM packages if the user has access to the service (LP #1883315)
    - Only display ESM Infra alerts if the distro is ESM

[Lucas Albuquerque Medeiros de Moura (lamoura)]

For this SRU bug I have created the following test script:

-----------------------------------
#!/bin/sh
set -x

series=hirsute
name=$series-dev

function update-notifier-ua-not-installed () {
    # Expects this output
    # x packages can be updated.
    # x of these updates are security updates.
    echo "-----------------------------------------------------------"
    echo "$series: ua not installed"
    lxc exec $name -- /usr/lib/update-notifier/apt-check --human-readable
    echo "-----------------------------------------------------------"
}

function update-notifier-ua-not-attached-esm () {
    # x updates can be installed immediately.
    # x of these updates are security updates.
    #
    # Enable UA Apps: ESM to receive x additional security updates.
    # See https://ubuntu.com/security/esm or run: sudo ua status
    echo "-----------------------------------------------------------"
    echo "$series esm: ua not attached"
    lxc exec $name -- /usr/lib/update-notifier/apt-check --human-readable
    echo "-----------------------------------------------------------"
}

function update-notifier-ua-not-attached () {
    # Expects this output
    # UA Infra: Extended Security Maintenance (ESM) is not enabled.
    #
    # x updates can be installed immediately.
    # x of these updates are security updates.
    echo "-----------------------------------------------------------"
    echo "$series non-esm: ua not attached"
    lxc exec $name -- /usr/lib/update-notifier/apt-check --human-readable
    echo "-----------------------------------------------------------"
}

function update-notifier-ua-attached-esm () {
    # Expects this output
    # x updates can be installed immediately.
    # x of these updates are fixed through UA Apps: ESM.
    # x of these updates are security updates.
    # To see these additional updates run: apt list --upgradable
    echo "-----------------------------------------------------------"
    echo "$series esm: ua attached"
    lxc exec $name -- /usr/lib/update-notifier/apt-check --human-readable
    echo "-----------------------------------------------------------"
}

function update-notifier-ua-attached () {
    # Expects this output
    # x updates can be installed immediately.
    # x of these updates are fixed through UA Apps: ESM.
    # x of these updates are security updates.
    # To see these additional updates run: apt list --upgradable
    echo "-----------------------------------------------------------"
    echo "$series non-esm: ua attached"
    lxc exec $name -- /usr/lib/update-notifier/apt-check --human-readable
    echo "-----------------------------------------------------------"
}

function turn-distro-into-esm-mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/is_esm_supported and is_not_currently_supported/True #comment/' /usr/lib/update-notifier/apt-check
}

function turn-distro-into-non-esm-mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/True #comment/False/' /usr/lib/update-notifier/apt-check
}

function setup-update-notifier() {
    lxc exec $name -- add-apt-repository ppa:lamoura/update-notifier...

[Brian Murray (brian-murray)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.40.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.30.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.192.1.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Lucas, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted update-notifier (3.192.30.7) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:20.04.10.6 (armhf, ppc64el, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#update-notifier

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lucas Albuquerque Medeiros de Moura (lamoura)]

I have tested this package for released xenial, bionic, focal and hirsute with the following script:

----------------------
#!/bin/sh
set -x

series=hirsute
name=$series-dev

function generate_motd_message() {
    message=$1
    echo "-----------------------------------------------------------"
    echo $message
    lxc exec $name -- /usr/lib/update-notifier/update-motd-updates-available --force
    lxc exec $name -- update-motd
    echo "-----------------------------------------------------------"
}

function turn_distro_into_esm_mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/is_esm_supported and is_not_currently_supported/True #comment/' /usr/lib/update-notifier/apt-check
}

function turn_distro_into_non_esm_mode() {
    # guarantee that xenial distro is on ESM mode
    lxc exec $name -- sed -i 's/True #comment/False/' /usr/lib/update-notifier/apt-check
}

function setup_update_notifier() {
# lxc exec $name -- add-apt-repository ppa:lamoura/update-notifier-test-ppa -y
# lxc exec $name -- apt update
# lxc exec $name -- apt install update-notifier-common -y
# lxc exec $name -- apt update
# lxc exec $name -- apt install python3-distro-info
# lxc file push data/apt_check.py $name/usr/lib/update-notifier/apt-check
    lxc exec $name -- sh -c "cat <<EOF >/etc/apt/sources.list.d/ubuntu-$series-proposed.list
    deb http://archive.ubuntu.com/ubuntu/ $series-proposed restricted main multiverse universe"
    lxc exec $name -- apt update
    lxc exec $name -- sh -c "apt install update-notifier -yq > /dev/null"
}

function install_ua() {
    lxc exec $name -- add-apt-repository ppa:ua-client/daily -y
    lxc exec $name -- sudo apt-get update
    lxc exec $name -- sudo apt-get install ubuntu-advantage-tools -y
    lxc exec $name -- ua version
    lxc exec $name -- sudo apt-get update
}

function ua_attach() {
    lxc exec $name -- sed -i 's/contracts.can/contracts.staging.can/' /etc/ubuntu-advantage/uaclient.conf
    lxc exec $name -- ua attach $UACLIENT_BEHAVE_CONTRACT_TOKEN_STAGING
}

function ua_detach() {
    lxc exec $name -- ua detach --assume-yes
}

function generate_ua_motd_messages() {
    lxc exec $name -- python3 /usr/lib/ubuntu-advantage/ua_update_messaging.py
}

function install_update_motd() {
    lxc exec $name -- apt install update-motd -yq
}

function turn_esm_apps_into_non_beta() {
    lxc exec $name -- sh -c "echo 'features:\n allow_beta: true' >> /etc/ubuntu-advantage/uaclient.conf"
}

function update_contract_effectiveto() {
    operation=$1
    num_days=$2
    replace_date=$(date -d "$date $operation$num_days days" +"%Y-%m-%dT00:00:00Z")
    echo $replace_date

    lxc exec $name -- sed -i "s/\"effectiveTo\": \"[^\"]*\"/\"effectiveTo\": \"$replace_date\"/g" /var/lib/ubuntu-advantage/private/machine-token.json
}

lxc delete --force $name
lxc launch ubuntu-daily:$series $name
sleep 10

setup_update_notifier
install_update_motd
generate_motd_message "$series: ua not installed"
install_ua
turn_esm_apps_into_non_beta
generate_ua_motd_messages
turn_distro_into_esm_mode
generate_motd_message "$series esm: ua not attached"
ua_attach
generate_ua_motd_messages
generate_motd_me...

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.41

---------------
update-notifier (3.192.41) impish; urgency=medium

  [ Lucas Moura ]

  * data/apt_check.py:
    - Add support to handle packages from ESM Apps in addition to ESM Infra
      and only display alerts if the distro is ESM. (LP: #1924766)
    - Do not display a count of ESM packages if the system does not have ESM
      enabled. (LP: #1883315)
    - Make distinction between standard security updates and ESM updates
      when performing package counts. (LP: #1926208)
    - use 'applied' instead of 'installed', redact 0 of these updates are
      security updates, and correct singular messages
  * debian/control: Add a dependency on python3-distro-info.

 -- Chad Smith <email address hidden> Thu, 22 Apr 2021 17:47:19 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.40.1

---------------
update-notifier (3.192.40.1) hirsute; urgency=medium

  [ Lucas Moura ]
  * data/apt_check.py:
    - Add support to handle packages from ESM Apps in addition to ESM Infra
      and only display alerts if the distro is ESM. (LP: #1924766)
    - Do not display a count of ESM packages if the system does not have ESM
      enabled. (LP: #1883315)
    - Make distinction between standard security updates and ESM updates
      when performing package counts. (LP: #1926208)
    - use 'applied' instead of 'installed', redact 0 of these updates are
      security updates, and correct singular messages
  * debian/control: Add a dependency on python3-distro-info.

 -- Chad Smith <email address hidden> Thu, 22 Apr 2021 17:47:19 -0600

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.30.7

---------------
update-notifier (3.192.30.7) focal; urgency=medium

  * data/apt_check.py:
    - Add support to handle packages from ESM Apps in addition to ESM Infra
      and only display alerts if the distro is ESM. (LP: #1924766)
    - Do not display a count of ESM packages if the system does not have ESM
      enabled. (LP: #1883315)
    - Make distinction between standard security updates and ESM updates
      when performing package counts. (LP: #1926208)
    - use 'applied' instead of 'installed', redact 0 of these updates are
      security updates, and correct singular messages
  * data/backend_helper.py:
    - fix pyflakes test
  * debian/control: Add a dependency on python3-distro-info.

 -- Lucas Moura <email address hidden> Thu, 22 Apr 2021 18:56:22 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.192.1.10

---------------
update-notifier (3.192.1.10) bionic; urgency=medium

  * data/apt_check.py:
    - Add support to handle packages from ESM Apps in addition to ESM Infra
      and only display alerts if the distro is ESM. (LP: #1924766)
    - Do not display a count of ESM packages if the system does not have ESM
      enabled. (LP: #1883315)
    - Make distinction between standard security updates and ESM updates
      when performing package counts. (LP: #1926208)
    - use 'applied' instead of 'installed', redact 0 of these updates are
      security updates, and correct singular messages
  * debian/control: Add a dependency on python3-distro-info.

 -- Lucas Moura <email address hidden> Thu, 22 Apr 2021 18:39:19 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.168.14

---------------
update-notifier (3.168.14) xenial; urgency=medium

  * data/apt_check.py:
    - Add support to handle packages from ESM Apps in addition to ESM Infra
      and only display alerts if the distro is ESM. (LP: #1924766)
    - Do not display a count of ESM packages if the system does not have ESM
      enabled. (LP: #1883315)
    - Make distinction betweem standard security updates and ESM updates
      when performing package counts. (LP: #1926208)
    - use 'applied' instead of 'installed', redact 0 of these updates are
      security updates, and correct singular messages
  * debian/control: Add a dependency on python3-distro-info.

 -- Lucas Moura <email address hidden> Tue, 20 Apr 2021 10:20:21 -0300