package leaves non-updated copy of /usr/sbin/apparmor_parser after update to apparmor-2.13.3-7ubuntu5.2. Orphaned older executable breaks docker

[Paul-Andre Panon]

There appears to be two copies of apparmor_parser installed by previous versions of the apparmor package, in /sbin and /usr/sbin. When updating the apparmor package to apparmor-2.13.3-7ubuntu5.2, only the /sbin/apparmor_parser executable is updated and the /usr/sbin copy is left unchanged. Being earlier the path, /usr/sbin/apparmor_parser is used by Docker when trying to register the docker-default apparmor profile for containers. The orphaned older executable reports a warning about a new parameter in the parser configuration file in the same package, and that warning breaks the version check that docker runs against that executable on the first line of output. trying to parse the warning while looking for the version number results in the error:

docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: strconv.Atoi: parsing "file": invalid syntax.

As a workaround, we've been replacing the old version in /usr/sbin with a symlink to the file in /sbin, but the package should be corrected to do appropriate behaviour (either delete the unnecessary(?) copy in /usr/sbin or replace it with a symlink)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Seth Arnold (seth-arnold)]

Hello Paul-Andre, I don't see any /usr/sbin/apparmor_parser files in any of the binary packages that I've got very easy access to:

sarnold@wopr:/dev/shm/apparmor $ find . -name apparmor_parser -ls
331800394 1472 -rwxr-xr-x 1 sarnold sarnold 1506552 Feb 28 14:18 ./apparmor_3.0.8-1ubuntu2/sbin/apparmor_parser
331800180 1472 -rwxr-xr-x 1 sarnold sarnold 1506552 Sep 23 2022 ./apparmor_3.0.7-1ubuntu2/sbin/apparmor_parser
331799966 1472 -rwxr-xr-x 1 sarnold sarnold 1506552 Nov 23 09:55 ./apparmor_3.0.7-1ubuntu2.1/sbin/apparmor_parser
331799752 1500 -rwxr-xr-x 1 sarnold sarnold 1535648 Mar 9 2022 ./apparmor_3.0.4-2ubuntu2/sbin/apparmor_parser
331799540 1508 -rwxr-xr-x 1 sarnold sarnold 1543872 Oct 19 2022 ./apparmor_3.0.4-2ubuntu2.2/sbin/apparmor_parser
331799361 832 -rwxr-xr-x 1 sarnold sarnold 849048 Apr 3 2014 ./apparmor_2.8.95~2430-0ubuntu5/sbin/apparmor_parser
331799175 1468 -rwxr-xr-x 1 sarnold sarnold 1501568 Apr 12 2020 ./apparmor_2.13.3-7ubuntu5/sbin/apparmor_parser
331798981 1488 -rwxr-xr-x 1 sarnold sarnold 1522176 Oct 10 2022 ./apparmor_2.13.3-7ubuntu5.2/sbin/apparmor_parser
331798786 1440 -rwxr-xr-x 1 sarnold sarnold 1472232 Apr 17 2018 ./apparmor_2.12-4ubuntu5/sbin/apparmor_parser
331798611 1440 -rwxr-xr-x 1 sarnold sarnold 1472232 Sep 27 2018 ./apparmor_2.12-4ubuntu5.1/sbin/apparmor_parser
331798311 1256 -rwxr-xr-x 1 sarnold sarnold 1282984 Apr 12 2016 ./apparmor_2.10.95-0ubuntu2/sbin/apparmor_parser
331798305 888 -rwxr-xr-x 1 sarnold sarnold 909192 Sep 27 2018 ./apparmor_2.10.95-0ubuntu2.6~14.04.4/sbin/apparmor_parser
331797891 1260 -rwxr-xr-x 1 sarnold sarnold 1287064 May 28 2019 ./apparmor_2.10.95-0ubuntu2.11/sbin/apparmor_parser
sarnold@wopr:/dev/shm/apparmor $ find . -name apparmor_parser -ls | grep usr
sarnold@wopr:/dev/shm/apparmor 1 $

On my focal and newer systems, /sbin is a symlink to /usr/sbin:

$ ls -ld /sbin /usr/sbin
lrwxrwxrwx 1 root root 8 Apr 10 2019 /sbin -> usr/sbin
drwxr-xr-x 2 root root 605 Apr 21 06:44 /usr/sbin

This is part of the usrmerge process: https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/

I'm curious how you've got a focal system where this isn't the case. How did this machine come to be? What's the broad outlines of its life history?

Thanks

[Paul-Andre Panon (ppanon-smtc)]

That's a good question. I'm not sure of the exact history of these systems, but I'm under the impression that they've gone through dist-upgrades from as least one older LTS and more likely at least 2 (i.e. from 16.04 to 18.04, to 20.04), but that predates my involvement. That said, when I looked at the ones prior to the upgrade, the file in /usr/sbin had the same May (I think) 2020 timestamp as the /sbin copy, so it perhaps was a hard link that failed to get upgraded by the package update.

[Seth Arnold (seth-arnold)]

Ah, interesting, TIL that focal is a bit odd here:

- we changed to usrmerge as default in the disco installer
- we added the 'usrmerge' package to convert old installations to this format in hirsute:
  https://bugs.launchpad.net/ubuntu/+source/usrmerge/+bug/1906671

So, if you installed with focal, you'd have the usrmerge filesystem setup.
If you initially installed with cosmic or earlier and upgrade, you won't get the usrmerge filesystem setup.

Your system is less strange than I thought; sadly, now I'm even more confused how you're seeing what you're seeing. These /sbin -> /usr/sbin symlinks are so awkward, it's easy to draw incorrect conclusions about what's going on, so be very careful before proceeding, but I expect you can delete the /usr/sbin/apparmor_parser if that is actually a symlink and hopefully never think of this again.

Be careful, of course, if you delete the only apparmor_parser on the system it'll be a pretty unhappy next reboot.

I'd double-check ls -l /sbin /usr/sbin /sbin/apparmor_parser /usr/sbin/apparmor_parser a few times before deciding what, if anything, to delete.

Thanks

[Paul-Andre Panon (ppanon-smtc)]

The /usr/sbin file not a symlink when it's causing an error. I'm guessing that it's a hard link but that the package upgrade is somehow deleting the /sbin directory entry (because it's the only one listed in the dpkg -L output) and building a new inode for the new file contents, but leaving the /usr/sbin entry pointing to the old inode. Our solution has been to delete the old version in /usr/sbin and replace it with a symlink, but this might affect more Docker users who run on release-updated systems. I did find some somewhat recent references to that strconv.Atoi error for docker users, so I expect others are having the same problem, which is why I'm reporting it here. Ideally you would create a package update that in the post-update script tests whether the /usr/sbin entry is a regular file/hard link or a symlink, and deletes the existing entry if it's not a symlink before recreating it as a symlink.

[Paul-Andre Panon (ppanon-smtc)]

Actually I just re-read and saw that you indicated /sbin was the symlink, (the reverse of what we did for our "fix") which seems really strange because dpkg -L shows /sbin/apparmor_parser as a file in the apparmor package, but not /usr/sbin.

ndco-admin@CARMD-EV-ALBLD7:~$ dpkg -L apparmor
/.
/etc
/etc/apparmor
/etc/apparmor/parser.conf
/etc/apparmor.d
/etc/apparmor.d/abi
/etc/apparmor.d/abi/2.13
/etc/apparmor.d/abstractions
/etc/apparmor.d/abstractions/X
/etc/apparmor.d/abstractions/apache2-common
/etc/apparmor.d/abstractions/apparmor_api
/etc/apparmor.d/abstractions/apparmor_api/change_profile
/etc/apparmor.d/abstractions/apparmor_api/examine
/etc/apparmor.d/abstractions/apparmor_api/find_mountpoint
/etc/apparmor.d/abstractions/apparmor_api/introspect
/etc/apparmor.d/abstractions/apparmor_api/is_enabled
/etc/apparmor.d/abstractions/aspell
/etc/apparmor.d/abstractions/audio
/etc/apparmor.d/abstractions/authentication
/etc/apparmor.d/abstractions/base
/etc/apparmor.d/abstractions/bash
/etc/apparmor.d/abstractions/consoles
/etc/apparmor.d/abstractions/cups-client
/etc/apparmor.d/abstractions/dbus
/etc/apparmor.d/abstractions/dbus-accessibility
/etc/apparmor.d/abstractions/dbus-accessibility-strict
/etc/apparmor.d/abstractions/dbus-session
/etc/apparmor.d/abstractions/dbus-session-strict
/etc/apparmor.d/abstractions/dbus-strict
/etc/apparmor.d/abstractions/dconf
/etc/apparmor.d/abstractions/dovecot-common
/etc/apparmor.d/abstractions/dri-common
/etc/apparmor.d/abstractions/dri-enumerate
/etc/apparmor.d/abstractions/enchant
/etc/apparmor.d/abstractions/fcitx
/etc/apparmor.d/abstractions/fcitx-strict
/etc/apparmor.d/abstractions/fonts
/etc/apparmor.d/abstractions/freedesktop.org
/etc/apparmor.d/abstractions/gnome
/etc/apparmor.d/abstractions/gnupg
/etc/apparmor.d/abstractions/ibus
/etc/apparmor.d/abstractions/kde
/etc/apparmor.d/abstractions/kde-globals-write
/etc/apparmor.d/abstractions/kde-icon-cache-write
/etc/apparmor.d/abstractions/kde-language-write
/etc/apparmor.d/abstractions/kerberosclient
/etc/apparmor.d/abstractions/ldapclient
/etc/apparmor.d/abstractions/libpam-systemd
/etc/apparmor.d/abstractions/likewise
/etc/apparmor.d/abstractions/mdns
/etc/apparmor.d/abstractions/mesa
/etc/apparmor.d/abstractions/mir
/etc/apparmor.d/abstractions/mozc
/etc/apparmor.d/abstractions/mysql
/etc/apparmor.d/abstractions/nameservice
/etc/apparmor.d/abstractions/nis
/etc/apparmor.d/abstractions/nvidia
/etc/apparmor.d/abstractions/opencl
/etc/apparmor.d/abstractions/opencl-common
/etc/apparmor.d/abstractions/opencl-intel
/etc/apparmor.d/abstractions/opencl-mesa
/etc/apparmor.d/abstractions/opencl-nvidia
/etc/apparmor.d/abstractions/opencl-pocl
/etc/apparmor.d/abstractions/openssl
/etc/apparmor.d/abstractions/orbit2
/etc/apparmor.d/abstractions/p11-kit
/etc/apparmor.d/abstractions/perl
/etc/apparmor.d/abstractions/php
/etc/apparmor.d/abstractions/php5
/etc/apparmor.d/abstractions/postfix-common
/etc/apparmor.d/abstractions/private-files
/etc/apparmor.d/abstractions/private-files-strict
/etc/apparmor.d/abstractions/python
/etc/apparmor.d/abstractions/qt5
/etc/apparmor.d/abstractions/qt5-compose-cache-write
/etc/apparmor.d/abstractions/qt5-settings-write
/etc/...

[Paul-Andre Panon (ppanon-smtc)]

Also
~$ dpkg -S /usr/sbin/apparmor_parser
dpkg-query: no path found matching pattern /usr/sbin/apparmor_parser
~$ dpkg -S /sbin/apparmor_parser
apparmor: /sbin/apparmor_parser

so it seems pretty messed up to me that the package file list would include the symbolic link but not the actual file. Something is very wrong there and leaving the package as-is is likely to cause more problems down the road. Also, as far as I know, the /sbin split with /usr/sbin is historically so that key functions are available in /sbin for emergency use when /usr is a separate partition and only the root partition is available because the /usr mount failed. In that context it doesn't seem to make a lot of sense to have /sbin/apparmor_parser be a symlink to /usr/sbin/apparmor_parser, because if you're trying to use it out of /sbin it's because /usr/sbin is unavailable. Either the actual file should be in /sbin because it's a critical tool that needs to be available if /usr doesn't mount, or else it should only be in /usr/sbin

I don't know what the right answer is here, but I've got a strong feeling the status quo is not it.

[Paul-Andre Panon (ppanon-smtc)]

Ah, I now see what you mean by the usrmerge. However I think that just reinforces my comments above. It doesn't make any sense for the installed package file to be installed in /sbin if /sbin is a symlink to /usr/sbin. if /usr/sbin is the canonical directory, and /sbin is an alias/symlink, then the apparmor_parser file should be installed in /usr/sbin.

Also, when it comes to Myth #5 and Myth #8 in that usrmerge page

Myth #5: Adopting the /usr merge in your distribution means additional work for your distribution's package maintainers

Fact: When the merge is implemented in other distributions and upstream, not adopting the /usr merge in your distribution means more work, adopting it is cheap.

Myth #8: The /usr merge will break my old installation which has /usr on a separate partition.

Fact: This is perfectly well supported, and one of the reasons we are actually doing this is to make placing /usr of a separate partition more thorough. What changes is simply that you need to boot with an initrd that mounts /usr before jumping into the root file system. Most distributions rely on initrds anyway, so effectively little changes.

Mm-hmm. Yeah,....

[Seth Arnold (seth-arnold)]

Your dpkg -S hits an ancient issue https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=134758

You're also exactly right about status quo being an unhappy place. Debian is currently trying to figure out a solution: https://lists.debian.org/debian-devel/2023/04/msg00008.html -- it's been in progress for years and probably will remain in progress for years.

I don't think the right answer is for individual packages to make changes -- Simon has enumerated some risks at: https://lists.debian.org/debian-devel/2023/04/msg00090.html

I don't know what the right answer is for your computer -- nor how you've even gotten into the situation you're in. I believe just blinding installing the usrmerge package to forcibly move all your executables and build symlinks would probably crash if you've got duplicate executables in both places.

My first thought to finding more collisions...

cd /bin ; echo * | tr ' ' '\n' > /tmp/bin
cd /usr/bin ; echo * | tr ' ' '\n' > /tmp/usrbin

comm -12 /tmp/bin /tmp/usrbin

cd /sbin ; echo * | tr ' ' '\n' > /tmp/sbin
cd /usr/sbin ; echo * | tr ' ' '\n' > /tmp/usrsbin

comm -12 /tmp/sbin /tmp/usrsbin

[Paul-Andre Panon (ppanon-smtc)]

OK, so basically this is due to the usrmerge change, that appears to be driven by Redhat who recommend reinstalls for each new major version. It was adopted by Debian and hence Uyuni and you haven't got a reliable way to apply the merge during the in-place upgrades that you historically supported; but it can randomly break things if you don't apply the usrmerge because packages from maintainers may randomly break things by making invalid assumptions about file deployments, but you won't do any work to fix those packages. You also appear to have failed to really document in the release notes that this is even a thing.

Sounds to me like you and Debian got hoodwinked by Redhat into breaking something that arguably wasn't broken.

That said, I do very much appreciate you taking the time to explain it and hopefully this discussion will be useful to others who have docker blow up for them due to the apparmor_parser as well.