apt-check python script doesn't catch issues before outputting bad data

[Kodiak Firesmith]

This type of error should be caught.

 /usr/lib/update-notifier/apt-check
E: Unknown Error: '<class 'KeyError'>' ("The cache has no package named 'libxmlb2:amd64'")

/usr/lib/update-notifier/apt-check --human-readable
E: Unknown Error: '<class 'KeyError'>' ("The cache has no package named 'libxmlb2:amd64'")

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kodiak Firesmith (kfiresmith-whoi)]

In the past 3+ weeks I've been playing whack-a-mole with this across a ~300 system fleet of servers.

Today a different package on a different host came up:

E: Unknown Error: '<class 'KeyError'>' ("The cache has no package named 'linux-headers-5.4.0-144-generic:amd64'")

Beyond the bug at hand, which of course should be fixed so that we aren't emitting unpredictable output from 'apt-check', whomever is in charge of Canonical repositories probably ought to be looking into these continuous sporadic cache problems. It's coming from the upstream cache we download AFAICT.

[Andrej Ricnik (a-ricnik)]

We've been seeing this, too, since 8 Feb.

https://askubuntu.com/questions/1457455/apt-check-python-apt-regression/1457480#1457480

[Kodiak Firesmith (kfiresmith-whoi)]

One of our python coders took a look at this issue with me today and had some observations that might help anyone trying to debug the issue:
https://stackoverflow.com/questions/75654312/why-do-the-two-python-apis-for-accessing-the-apt-cache-return-different-sets-of

[Kodiak Firesmith (kfiresmith-whoi)]

This seems like the sort of oddity that Julian Klode might be interested in. Not sure about etiquette for bringing someone's attention to a bug so I apologize Julian, if subscribing you wasn't the right move.

[David A. Desrosiers (setuid)]

Kodiak, I've looped Julian in, internally... when he comes back online, he'll weigh in. I've given him the internal and external links and context to sort through this. Thanks for being patient while we iterate through the various options.

[Julian Andres Klode (juliank)]

@kodiak apt.Cache only includes real packages, the one in apt_pkg also has virtual packages. You can see that in apt/cache.py how it filters the apt_pkg.Cache.

[Kodiak Firesmith (kfiresmith-whoi)]

One thing to note is that while this problem is present in jammy-updates and focal-updates version of apt_check.py in /src, the version already in the devel branch seems to fix the issue.

( This works: https://git.launchpad.net/ubuntu/+source/update-notifier/tree/data/apt_check.py,
these don't:
 - https://git.launchpad.net/ubuntu/+source/update-notifier/tree/data/apt_check.py?h=ubuntu%2Ffocal-updates
 - https://git.launchpad.net/ubuntu/+source/update-notifier/tree/data/apt_check.py?h=ubuntu%2Fjammy-updates
)

Due to what was added in this commit:
update-notifier (3.192.63) lunar; urgency=medium

  * Isolate creation of the esm apt cache in apt-check (LP: #2008212)

 -- Renan Rodrigo Barbosa <email address hidden> Mon, 27 Feb 2023 23:15:02 -0300

+def get_apt_pkg_esm_cache():
+ """Get an apt_pkg cache with the ubuntu-advantage-tools esm data.
+
+ Set the configuration to get the u-a-t cache, then set it back to an
+ empty configuration state and init again so other calls to Cache work as
+ expected.
+ """
+ for key in apt_pkg.config.keys():
+ if re.match(r"^Acquire", key) is None:
+ apt_pkg.config.clear(key)
+ apt_pkg.config.set("Dir", PRO_ESM_CACHE_DIR)
+ apt_pkg.init()
+ try:
+ esm_cache = apt_pkg.Cache(progress=None)
+ except apt_pkg.Error:
+ esm_cache = None
+
+ for key in apt_pkg.config.keys():
+ apt_pkg.config.clear(key)
+ apt_pkg.init()
+
+ return esm_cache
+
+
 def init():
     " init the system, be nice "
     # FIXME: do a ionice here too?
@@ -365,12 +390,7 @@ def run(options=None):
         sys.stderr.write("E: " + _("Error: Marking the upgrade (%s)") % e)
         sys.exit(-1)

- try:
- apt_pkg.config.set("Dir", "/var/lib/ubuntu-advantage/apt-esm/")
- apt_pkg.init_system()
- esm_cache = apt_pkg.Cache(progress=None)
- except apt_pkg.Error:
- esm_cache = None
+ esm_cache = get_apt_pkg_esm_cache()

[Renan Rodrigo (renanrodrigo)]

Interesting that this got fixed: the thing is that because the esm-cache was changing the global configuration, any further calls to the cache would have missed packages that do exist.

The version which contains this new commit was already uploaded to all LTS+lunar, and now sits unapproved waiting for a SRU person to pick it. Soon it should be in xbfj-updates.

[Mauricio Faria de Oliveira (mfo)]

Sorry for the noise.

Thanks, Renan and Kodiak.

Marking this bug as duplicate of bug 2008212, where the SRUs are being tracked/updates posted.

[Kodiak Firesmith (kfiresmith-whoi)]

Hello Mauricio & Renan,
Thanks so much for the fix, much appreciated.