unattended-upgrade ignores apt-pinning to not-allowed origins

[Piotr Henryk Dabrowski]

unattended-upgrade ignores apt-pinning to not-allowed origins
=============================================================

BUG:

unattended-upgrade notices an upgrade available in NOT-ALLOWED ORIGINS,
but then completely ignores those repositories
even if they contain apt-pinned versions
that are more favorable than versions from allowed origins.

The situation repeats every time there is an upgrade available in an external PPA.

SOLUTION:

unattended-upgrade should not ignore NOT ALLOWED ORIGINS,
but check them for providing more favorable version
and in such case restrain from doing ANY upgrades for such packages.

Instructions for ubuntu lunar 23.04:
------------------------------------

0. Upgrade all packages, uninstall Firefox:

    $ sudo apt update

    $ sudo apt upgrade

    $ sudo snap remove firefox

    $ sudo apt remove firefox

    $ apt-cache policy firefox
    firefox:
      Installed: (none)
      Candidate: 1:1snap1-0ubuntu3
      Version table:
         1:1snap1-0ubuntu3 500
            500 http://pl.archive.ubuntu.com/ubuntu lunar/main arm64 Packages

1. Add mozilla-team Firefox PPA and apt-pin it with priority 1001:

    $ echo 'deb https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar main' | sudo tee /etc/apt/sources.list.d/firefox.list

    $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9BDB3D89CE49EC21

    $ echo -e 'Package: *\nPin: release o=LP-PPA-mozillateam\nPin-Priority: 1001' | sudo tee /etc/apt/preferences.d/firefox

    $ sudo apt update

    $ apt-cache policy firefox
    firefox:
      Installed: (none)
      Candidate: 117.0+build2-0ubuntu0.23.04.1~mt1
      Version table:
         1:1snap1-0ubuntu3 500
            500 http://pl.archive.ubuntu.com/ubuntu lunar/main arm64 Packages
         117.0+build2-0ubuntu0.23.04.1~mt1 1001
           1001 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 Packages

2. Install Firefox (from mozilla-team Firefox PPA, as pinned):

    $ sudo apt install firefox

    $ apt-cache policy firefox
    firefox:
      Installed: 117.0+build2-0ubuntu0.23.04.1~mt1
      Candidate: 117.0+build2-0ubuntu0.23.04.1~mt1
      Version table:
         1:1snap1-0ubuntu3 500
            500 http://pl.archive.ubuntu.com/ubuntu lunar/main arm64 Packages
     *** 117.0+build2-0ubuntu0.23.04.1~mt1 1001
           1001 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 Packages
            100 /var/lib/dpkg/status

3. SIMULATE AVAILABLE UPGRADE by downgrading Firefox from Mozilla-Team's version for Ubuntu 23.04 to 22.04:

Download .deb. file from Mozilla-Team's PPA:

    $ wget $(apt-get download --print-uris firefox | cut -d' ' -f1 | tr -d "'" | sed -E 's/0ubuntu0\.[0-9]+\.[0-9]+\./0ubuntu0.22.04./')

Install it:

    $ sudo dpkg -i firefox_*.22.04.*.deb
    dpkg: warning: downgrading firefox from 117.0+build2-0ubuntu0.23.04.1~mt1 to 117.0+build2-0ubuntu0.22.04.1~mt1
    (Reading database ... 295244 files and directories currently installed.)
    Preparing to unpack firefox_117.0+build2-0ubuntu0.22.04.1~mt1_arm64.deb ...
    Unpacking firefox (117.0+build2-0ubuntu0.22.04.1~mt1) over (117.0+build2-0ubuntu0.23.04.1~mt1) ...
    Setting up firefox (117.0+build2-0ubuntu0.22.04.1~mt1) ...
    Please restart all running instances of firefox, or you will experience problems.
    Processing triggers for gnome-menus (3.36.0-1.1ubuntu1) ...
    Processing triggers for desktop-file-utils (0.26-1ubuntu5) ...
    Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
    Processing triggers for hicolor-icon-theme (0.17-2) ...
    Processing triggers for man-db (2.11.2-1) ...

    $ apt-cache policy firefox
    firefox:
      Installed: 117.0+build2-0ubuntu0.22.04.1~mt1
      Candidate: 117.0+build2-0ubuntu0.23.04.1~mt1
      Version table:
         1:1snap1-0ubuntu3 500
            500 http://pl.archive.ubuntu.com/ubuntu lunar/main arm64 Packages
         117.0+build2-0ubuntu0.23.04.1~mt1 1001
           1001 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 Packages
     *** 117.0+build2-0ubuntu0.22.04.1~mt1 100
            100 /var/lib/dpkg/status

4. Bug in unattended-upgrade:

Firefox is now at priority 100 ("now").
Firefox snap package is at priority 500.
Mozilla-Team PPA has priority 1001, BUT IS NOT IN UNATTENDED-UPGRADE'S "ALLOWED ORIGINS".

BUG: unattended-upgrade upgrades Firefox package to 1:1snap1-0ubuntu3:

    $ sudo unattended-upgrade -v
    Starting unattended upgrades script
    Allowed origins are: o=Ubuntu,a=lunar, o=Ubuntu,a=lunar-security, o=UbuntuESMApps,a=lunar-apps-security, o=UbuntuESM,a=lunar-infra-security
    Initial blacklist:
    Initial whitelist (not strict):
    Packages that will be upgraded: firefox
    Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
    Preconfiguring packages ...
    Preconfiguring packages ...
    (Reading database ... 295244 files and directories currently installed.)
    Preparing to unpack .../firefox_1%3a1snap1-0ubuntu3_arm64.deb ...
    => Installing the firefox snap
    ==> Checking connectivity with the snap store
    ==> Installing the firefox snap
    => Snap installation complete
    Unpacking firefox (1:1snap1-0ubuntu3) over (117.0+build2-0ubuntu0.22.04.1~mt1) ...
    dpkg: warning: unable to delete old directory '/etc/firefox': Directory not empty
    dpkg: warning: unable to delete old directory '/etc/apport/blacklist.d': Directory not empty
    Setting up firefox (1:1snap1-0ubuntu3) ...
    Removing obsolete conffile /etc/firefox/syspref.js ...
    Processing triggers for man-db (2.11.2-1) ...
    Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
    Processing triggers for desktop-file-utils (0.26-1ubuntu5) ...
    Processing triggers for hicolor-icon-theme (0.17-2) ...
    Processing triggers for gnome-menus (3.36.0-1.1ubuntu1) ...
    All upgrades installed

5. However apt will now (properly) want to downgrade Firefox to the version from PPA:

    $ sudo apt upgrade
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    Calculating upgrade... Done
    #
    # You can verify the status of security fixes using the `pro fix` command.
    # E.g., a recent Ruby vulnerability can be checked with: `pro fix USN-6219-1`
    # For more detail see: https://ubuntu.com/security/notices/USN-6219-1
    #
    The following packages will be DOWNGRADED:
      firefox
    0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
    Need to get 59.3 MB of archives.
    After this operation, 216 MB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 firefox arm64 117.0+build2-0ubuntu0.23.04.1~mt1 [59.3 MB]
    Fetched 59.3 MB in 1s (45.4 MB/s)
    dpkg: warning: downgrading firefox from 1:1snap1-0ubuntu3 to 117.0+build2-0ubuntu0.23.04.1~mt1
    (Reading database ... 295166 files and directories currently installed.)
    Preparing to unpack .../firefox_117.0+build2-0ubuntu0.23.04.1~mt1_arm64.deb ...
    Unpacking firefox (117.0+build2-0ubuntu0.23.04.1~mt1) over (1:1snap1-0ubuntu3) ...
    Setting up firefox (117.0+build2-0ubuntu0.23.04.1~mt1) ...
    Please restart all running instances of firefox, or you will experience problems.
    Processing triggers for man-db (2.11.2-1) ...
    Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
    Processing triggers for desktop-file-utils (0.26-1ubuntu5) ...
    Processing triggers for hicolor-icon-theme (0.17-2) ...
    Processing triggers for gnome-menus (3.36.0-1.1ubuntu1) ...

The situation repeats every time there is an upgrade available in an external PPA.

6. Conclusion:

unattended-upgrade should not ignore NOT ALLOWED ORIGINS,
but check them for providing more favorable version
and in such case restrain from doing ANY upgrades for such packages.

WORKAROUND
----------

A. Go back to mozilla-team's 22.04 deb:

    $ sudo dpkg -i firefox_*.22.04.*.deb

    $ sudo snap remove firefox

B. Pin Ubuntu's official version to 1:

    $ echo -e 'Package: firefox\nPin: release o=Ubuntu\nPin-Priority: 1' | sudo tee /etc/apt/preferences.d/firefox-workaround

    $ apt-cache policy firefox
    firefox:
      Installed: 117.0+build2-0ubuntu0.22.04.1~mt1
      Candidate: 117.0+build2-0ubuntu0.23.04.1~mt1
      Version table:
         1:1snap1-0ubuntu3 1
            500 http://pl.archive.ubuntu.com/ubuntu lunar/main arm64 Packages
         117.0+build2-0ubuntu0.23.04.1~mt1 1001
           1001 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 Packages
     *** 117.0+build2-0ubuntu0.22.04.1~mt1 100
            100 /var/lib/dpkg/status

C. Priority 1 < 100, so Firefox will NOT be upgraded to 1:1snap1-0ubuntu3

    $ sudo unattended-upgrade -v
    Starting unattended upgrades script
    Allowed origins are: o=Ubuntu,a=lunar, o=Ubuntu,a=lunar-security, o=UbuntuESMApps,a=lunar-apps-security, o=UbuntuESM,a=lunar-infra-security
    Initial blacklist:
    Initial whitelist (not strict):
    MarkUpgrade() called on a non-upgradeable pkg: 'firefox'
    No packages found that can be upgraded unattended and no pending auto-removals

D. apt will upgrade Firefox to the latest Mozilla-Team version:

    $ sudo apt upgrade
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    Calculating upgrade... Done
    #
    # You can verify the status of security fixes using the `pro fix` command.
    # E.g., a recent Ruby vulnerability can be checked with: `pro fix USN-6219-1`
    # For more detail see: https://ubuntu.com/security/notices/USN-6219-1
    #
    The following packages will be upgraded:
      firefox
    1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Need to get 59.3 MB of archives.
    After this operation, 751 kB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar/main arm64 firefox arm64 117.0+build2-0ubuntu0.23.04.1~mt1 [59.3 MB]
    Fetched 5171 kB in 0s (14.0 MB/s)
    (Reading database ... 295244 files and directories currently installed.)
    Preparing to unpack .../firefox_117.0+build2-0ubuntu0.23.04.1~mt1_arm64.deb ...
    Unpacking firefox (117.0+build2-0ubuntu0.23.04.1~mt1) over (117.0+build2-0ubuntu0.22.04.1~mt1) ...
    Setting up firefox (117.0+build2-0ubuntu0.23.04.1~mt1) ...
    Please restart all running instances of firefox, or you will experience problems.
    Processing triggers for man-db (2.11.2-1) ...
    Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
    Processing triggers for desktop-file-utils (0.26-1ubuntu5) ...
    Processing triggers for hicolor-icon-theme (0.17-2) ...
    Processing triggers for gnome-menus (3.36.0-1.1ubuntu1) ...

[Piotr Henryk Dabrowski (phd)]

Bug https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1999308 is most likely caused by this issue.

[Piotr Henryk Dabrowski (phd)]

Confirmed by https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1999308

[Julian Andres Klode (juliank)]

I think this is more a misunderstanding here, but the "workaround" is the right solution for this issue.

Specifically, unattended-upgrades is exactly designed to give you less preferable upgrades because the others are reserved for manual upgrades (i.e. -security vs -updates). That is, the security update should still be installed and not ignored because there is a higher version available in -updates for manual installing.

The same principle applies here: You configured a firefox PPA for manual upgrades, but there is an "intermediate" upgrade available that you did not configure.

Now of course this is a particularly odd case because the intermediate update would block the prefered update from being installed (due to the epoch), and while one could fix that particular corner case, it has problems:

- it's very hard to implement: not allowed origins are pinned to the minimum, we never see that they'd otherwise be the candidate.
- it's inconsistent with pinning for other intermediate packages that will get upgraded (i.e. a version 1 when you have version 2 pinned up)

[Jan I (ifi-jani)]

@juliank - the whole point of using the Firefox package repo is to get timely security updates, which are regrettably impossible with Ubuntu's current snap-based implementation.

Perhaps this should be reassigned as an issue for whatever reinstalls the firefox transitional package, which should not magically reinstall itself every once in a while?

[Piotr Henryk Dabrowski (phd)]

@juliank

I think you missed the point of this bug report and its grave status for "unattended-upgrades".

This bug happens even when there is *no* "intermediate" update available in the Ubuntu repository (allowed-origin).
When there is an *UPDATE AVAILABLE IN AN EXTERNAL REPOSITORY* (not-allowed-origin),
"unattended-upgrades" installs the package from the Ubuntu repository, which hasn't even changed (!)

Example:
    ubuntu: 1:1.0
    now: 0:123.0
    ppa: 0:123.0 (apt-pinned to 1001)
Update available *IN THE EXTERNAL REPOSITORY*:
    ubuntu: 1:1.0
    now: 0:123.0
    ppa: 0:124.0 (apt-pinned to 1001)
"unattended-upgrades" stupidly installs 1:1.0 (WTF?!):
    ubuntu: 1:1.0
    now: 1:1.0
    ppa: 0:124.0 (apt-pinned to 1001)
And then a manual "apt-get upgrade" (properly) installs 0:124.0:
    ubuntu: 1:1.0
    now: 0:124.0
    ppa: 0:124.0 (apt-pinned to 1001)

> but the "workaround" is the right solution for this issue.

(workaround = pinning Ubuntu repository to 1/-1 for all external packages)
Wrong.
You may add the entire Debian repository (which will be not-allowed-origin),
and now you have to pin every package (and dependency!) installed from Debian *by name*
in order for "unattended-upgrades" not to mess with them.
This "solution" would be ridiculous.

> particularly odd case

No odd case here: there was *no* update available in the Ubuntu repository in this case.
It's "unattended-upgrades" behavior that is odd and completely broken.

> it's very hard to implement

This doesn't justify keeping a grave bug in "unattended-upgrades" behavior.

And it shouldn't be that hard:
1. Get the list of packages+version+origin that can be updated no matter of the allowed-origin state, no messing with apt-pinning.
2. Upgrade only the packages from this list that have origins that are allowed.

To sum up: "unattended-upgrades" completely breaks APT pinning (meant mostly for external repositories) and selects completely wrong versions for installation candidates. Period.

[Piotr Henryk Dabrowski (phd)]

Please reopen.

[Piotr Henryk Dabrowski (phd)]

Upstream issue predating this report: https://github.com/mvo5/unattended-upgrades/issues/319

[Julian Andres Klode (juliank)]

As explained before, the behavior here is intended and not a bug.

@Jan You are free to configure unattended-upgrades to install from the PPA or add negative pins for the Ubuntu (and UbuntuESM archives, if enabled) for src:firefox to prevent the snap from being installed.

@Piotr I will not correspond to the complete message but

> (workaround = pinning Ubuntu repository to 1/-1 for all external packages)
> Wrong.
> You may add the entire Debian repository (which will be not-allowed-origin),
> and now you have to pin every package (and dependency!) installed from Debian *by name*
> in order for "unattended-upgrades" not to mess with them.
>This "solution" would be ridiculous.

I stand by what I said. If you don't want firefox from the official repository, pin it down. You'll want

Package: src:firefox
Pin: release o=Ubuntu
Pin-Priority: -1

Package: src:firefox
Pin: release o=UbuntuESM
Pin-Priority: -1

Of course if you install multiple packages like that you need to list them all in the package line.

This is noticably different from pinning up the PPA and again there is no easy way out here, the behaviour as is is what is generally needed.

Adding the Debian repository is *NOT* supported.

[Julian Andres Klode (juliank)]

Once APT implements origin tracking this should no longer be a problem, or it might be; generally speaking the idea is that if you install a package from one archive, APT doesn't switch it. You can also configure rules of `o=Ubuntu -> o=UbuntuESM` to allow transitions between archives.

However one idea there is to add a rule `* -> o=Ubuntu`, `* -> o=UbuntuESM`; that is that every PPA package can be replaced by a newer Ubuntu package. Again usually that is what you want, you added a PPA, you upgrade your Ubuntu, Ubuntu now has that package so you don't need the PPA anymore and should not be sticking to it. But if you were to remove the rules you'd get the right behavior.

My concern is really more people adding hundreds of PPAs and getting undefined mixes between them, as APT will pick the highest version of a given package in any PPA or the Ubuntu archive. So now you if you install foo and libbar from ppa:foobar and then add ppa:moobar it doesn't replace libbar unless you ask it to.