20apt-esm-hook.conf is missing semicolons at end of option

[Brian Murray]

[Impact]

This bug caused aptdaemon to miss two configuration options that ubuntu-advantage-tools relies on for updating its messaging based on apt state. Apt itself parsed the options despite the syntax mistake, which led to a difference in understanding between apt and aptdaemon. This difference caused autopkgtest failures.

Further, because aptdaemon could not parse these two options, it may result in operations triggered through aptdaemon not properly updating ubuntu advantage esm messaging. This means ubuntu-advantage-tools may print confusing or out of date messages related to esm in some cases.

The fix is to add the semicolons required in the apt config syntax that we missed. This fix allows aptdaemon to parse all of our config settings, and means apt and aptdaemon will have a shared understanding of the configuration again.

[Test Plan]

These test steps assume that the proposed version of ua to release is available in ppa:ua-client/staging - check with the ubuntu-advantage-tools team before proceeding to make sure this is the case.

Start a container of an affected release (tested in xenial, but should work on any affected release). Inside the container:
```
sudo apt update
sudo apt install aptdaemon
```

Now run the following python3 script
```
from aptdaemon import config
c = config.ConfigWriter()
uataptconf = open("/etc/apt/apt.conf.d/20apt-esm-hook.conf")
lines = uataptconf.readlines()
print(c.parse(lines).keys())
uataptconf.close()
```

You should see output like the following:
```
dict_keys(['apt::install::post-invoke-success', 'apt::update::post-invoke-stats'])
```
Note that the above only includes 2 out of the 4 intended config settings from our config file.

Now install the fixed version:
```
sudo add-apt-repository ppa:ua-client/staging
sudo apt-get update
sudo apt install ubuntu-advantage-tools
```

And run the above python script again. You should see output like the following:
```
dict_keys(['apt::install::pre-invoke', 'apt::install::post-invoke-success', 'apt::update::post-invoke-stats', 'aptcli::hooks::upgrade'])
```

Notice that it now finds all of the intended config settings.

[Where problems could occur]

The fix is adding two semicolons to the apt config file. The only problem that could occur is that, if a mistake was made, a similar apt config parsing problem could occur. Perhaps if this caused additional config parsing problems, then our message updating hooks would be broken in a different way from this bug. This would result in lack of, or out-of-date, messaging about esm packages from ubuntu-advantage-tools.

[Other Info]

A hotfix was already released to impish as 27.1.1 to stop the autopkgtest failures and unblock work that was interrupted by this bug. It successfully solved the issue.

[Original Description]

Recently aptdaemon started having autopkgtest failures with its tests.test_configparser.ConfigurationParserTestCase test. The failure follows:

Test if credentials of repositories are store securely in a ... ok

======================================================================
ERROR: test_ (tests.test_configparser.ConfigurationParserTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.cp3vHc/build.adU/src/tests/test_configparser.py", line 82, in test_
    config[key.lower()]]
KeyError: 'apt::install::pre-invoke'

----------------------------------------------------------------------
Ran 79 tests in 109.732s

FAILED (errors=1, skipped=21)
Test failed: <unittest.runner.TextTestResult run=79 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=79 errors=1 failures=0>

The "APT::Install::Pre-Invoke" key is provided by /etc/apt/apt.conf.d/20apt-esm-hook.conf which is from ubuntu-advantage-tools, so I'd guess this started happening when it moved to -updates (on 20210513). This would also explain why it is affecting every release.

[Julian Andres Klode (juliank)]

ubuntu-advantage-tools has incorrect syntax in its apt.conf.d snippet; missing the semicolon at the end of the options for the new options introduced recently:

APT::Install::Pre-Invoke {
        "[ ! -f /usr/lib/ubuntu-advantage/apt-esm-hook ] || /usr/lib/ubuntu-advantage/apt-esm-hook pre-invoke || true";
}

AptCli::Hooks::Upgrade {
        "[ ! -f /usr/lib/ubuntu-advantage/apt-esm-json-hook ] || /usr/lib/ubuntu-advantage/apt-esm-json-hook || true";
}

This causes aptdaemon's parser to stumble over it; while apt's is too badly written to notice :)

[Dimitri John Ledkov (xnox)]

https://github.com/canonical/ubuntu-advantage-client/pull/1681/files

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.1~21.10.1.1

---------------
ubuntu-advantage-tools (27.1~21.10.1.1) impish; urgency=medium

  * Cherrypick upstream pr #1681 to unbreak many migrations. LP: #1930741

 -- Dimitri John Ledkov <email address hidden> Wed, 16 Jun 2021 17:14:08 +0100

[Lucas Albuquerque Medeiros de Moura (lamoura)]

I can confirm that this bug is fixed in the proposed 27.2.1 package on Xenial, Bionic, Focal and Hirsute by using the following script:

----------------------
#!/bin/sh
set -x

series=xenial
name=$series-check

lxc delete --force $name
lxc launch ubuntu-daily:$series $name

lxc exec $name -- sudo sh -c "cat <<EOF >/etc/apt/sources.list.d/ubuntu-$series-proposed.list
## Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $series-proposed restricted main multiverse universe
EOF"
sleep 10
lxc exec $name -- sh -c "sudo apt-get update > /dev/null"
lxc exec $name -- sh -c "sudo apt-get install aptdaemon -yq > /dev/null"
# Verify that the output shows only:
# dict_keys(['apt::install::post-invoke-success', 'apt::update::post-invoke-stats'])
lxc exec $name -- python3 -c "
from aptdaemon import config
c = config.ConfigWriter()
uataptconf = open('/etc/apt/apt.conf.d/20apt-esm-hook.conf')
lines = uataptconf.readlines()
print(c.parse(lines).keys())
uataptconf.close()
"

lxc exec $name -- sh -c "sudo apt-get install ubuntu-advantage-tools -yq > /dev/null"
lxc exec $name -- ua version

# Verify now that the output shows:
# dict_keys(['apt::install::pre-invoke', 'apt::install::post-invoke-success',
# 'apt::update::post-invoke-stats', 'aptcli::hooks::upgrade'])
lxc exec $name -- python3 -c "
from aptdaemon import config
c = config.ConfigWriter()
uataptconf = open('/etc/apt/apt.conf.d/20apt-esm-hook.conf')
lines = uataptconf.readlines()
print(c.parse(lines).keys())
uataptconf.close()
"

set +x
--------------------------------

This issue is already fixed on impish since release 27.1~21.10.1.1

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.2.1~21.04.1

---------------
ubuntu-advantage-tools (27.2.1~21.04.1) hirsute; urgency=medium

  * Backport new upstream release: (LP: #1934902) to hirsute

 -- Chad Smith <email address hidden> Fri, 16 Jul 2021 12:49:23 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.2.1~20.04.1

---------------
ubuntu-advantage-tools (27.2.1~20.04.1) focal; urgency=medium

  * Backport new upstream release: (LP: #1934902) to focal

 -- Chad Smith <email address hidden> Fri, 16 Jul 2021 12:48:51 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.2.1~18.04.1

---------------
ubuntu-advantage-tools (27.2.1~18.04.1) bionic; urgency=medium

  * Backport new upstream release: (LP: #1934902) to bionic

 -- Chad Smith <email address hidden> Fri, 16 Jul 2021 12:48:22 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 27.2.1~16.04.1

---------------
ubuntu-advantage-tools (27.2.1~16.04.1) xenial; urgency=medium

  * Backport new upstream release: (LP: #1934902) to xenial

 -- Chad Smith <email address hidden> Fri, 16 Jul 2021 12:47:53 -0600