security updates with a new dependency don't get installed

[Jarno Suni]

Test Case
---------
1) Boot a xenial system w/o chromium browser and w/o libspeechd2 installed
2) Install the release version of chromium browser e.g. "sudo apt-get install chromium-browser=49.0.2623.108-0ubuntu1.1233 chromium-browser-l10n=49.0.2623.108-0ubuntu1.1233 chromium-codecs-ffmpeg-extra=49.0.2623.108-0ubuntu1.1233"
3) Run apt-get update if you didn't already
4) Run "sudo /usr/bin/unattended-upgrades -v -d"
5) Observe the following output "Checking: chromium-browser ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libspeechd2' not in allowed origin
sanity check failed"

With the version of unattended-upgrades from -proposed libspeechd2 should be from an allowed origin and chromium-browser will get updated.

Regression Potential
--------------------
This change modifies the behavior of unattended-upgrades such that new packages will be installed on a user's system and they may not except such behavior (e.g. why was libspeechd2 insalled?). However, this seems better than not installing security updates and leaving people's systems vulnerable to attack.

Original Description
--------------------
E.g. chromium-browser has an update, but U-U does not update it. I saw in update-manager that the security update is available before running U-U. Afterwards I can install the update by update-manager.

ProblemType: BugDistroRelease: Ubuntu 16.04
Package: unattended-upgrades 0.90
ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16
Uname: Linux 4.4.0-36-generic i686
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: i386
CurrentDesktop: XFCE
Date: Sat Sep 17 11:13:40 2016
InstallationDate: Installed on 2016-09-05 (11 days ago)
InstallationMedia: Mythbuntu 16.04.1 LTS "Xenial Xerus" - Release i386 (20160719)
PackageArchitecture: allSourcePackage: unattended-upgrades
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Murray (brian-murray)]

Here's the output of unattended-upgrades run with --debug:

bdmurray@flash:~$ sudo /usr/bin/unattended-upgrade --verbose --debug
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: ['o=Ubuntu,a=xenial-security', 'o=Ubuntu,a=xenial-updates']
Checking: chromium-browser ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libspeechd2' not in allowed origin
sanity check failed
Checking: chromium-browser-l10n ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libspeechd2' not in allowed origin
sanity check failed
Checking: chromium-codecs-ffmpeg-extra ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libspeechd2' not in allowed origin
sanity check failed
Checking: mysql-server-5.7 ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'main' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libevent-core-2.0-5' not in allowed origin
sanity check failed
Checking: mysql-server-core-5.7 ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'main' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
pkg 'libevent-core-2.0-5' not in allowed origin
sanity check failed
pkgs that look like they should be upgraded:
Fetched 0 B in 0s (0 B/s)
fetch.run() result: 0
blacklist: []
whitelist: []
Packages that will be upgraded:
InstCount=0 DelCount=0 BrokenCount=0
Extracting content from '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log' since '2016-09-21 16:12:17'

[Brian Murray (brian-murray)]

Do a dist-upgrade we can see that some new packages are going to be installed.

bdmurray@flash:~$ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libevent-core-2.0-5 libspeechd2
The following packages will be upgraded:
  chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra mysql-server-5.7 mysql-server-core-5.7
5 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/68.3 MB of archives.
After this operation, 52.0 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

This is causing the issue because they are in main while the packages to upgrade are in universe.

[Brian Murray (brian-murray)]

Modifying 50unattended-upgrades in /etc/apt/apt.conf.d/ to include the following line allowed the unattended-upgrade to happen.

        "${distro_id}:${distro_codename}";

[Brian Murray (brian-murray)]

Although unattended-upgrades didn't say anything about the new packages being installed...

bdmurray@flash:~$ sudo /usr/bin/unattended-upgrade --verbose --dry-run
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security', 'o=Ubuntu,a=xenial-updates']
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra mysql-server-5.7 mysql-server-core-5.7
Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'

[Brian Murray (brian-murray)]

With regards to comment 3 this is happening because the packages in -security or -updates added a dependency on a package which it did not previously have and the dependent package is in the release pocket which isn't an allowed origin.

[Seth Arnold (seth-arnold)]

Is it possible to relax these rules without also bringing in apt-get dist-upgrade's unfortunate ability to uninstall packages when it thinks that's the shortest solution? We've seen cases where dist-upgrade sometimes tries to remove sudo or the signed shim.

Thanks

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.92ubuntu1

---------------
unattended-upgrades (0.92ubuntu1) yakkety; urgency=medium

  * Add debian/patches/enable-release-pocket.patch which sets the release
    pocket as an allowed origin so that security updates with a new dependency
    will be upgraded. (LP: #1624641)

 -- Brian Murray <email address hidden> Tue, 11 Oct 2016 08:36:53 -0700

[Martin Pitt (pitti)]

Hello Jarno, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

bdmurray@clean-xenial-amd64:~$ apt-cache policy unattended-upgrades
unattended-upgrades:
  Installed: 0.90ubuntu0.1
  Candidate: 0.90ubuntu0.1
  Version table:
 *** 0.90ubuntu0.1 100
        100 /var/lib/dpkg/status
     0.90 500
        500 http://192.168.10.7/ubuntu xenial/main amd64 Packages
        500 http://192.168.10.7/ubuntu xenial/main i386 Packages

2016-10-27 13:09:51,158 DEBUG Checking: chromium-browser ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
2016-10-27 13:09:52,224 DEBUG Checking: chromium-browser-l10n ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
2016-10-27 13:09:53,209 DEBUG Checking: chromium-codecs-ffmpeg-extra ([<Origin component:'universe' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>, <Origin component:'universe' archive:'xenial-security' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
2016-10-27 13:09:54,990 DEBUG Checking: snap-confine ([<Origin component:'main' archive:'xenial-updates' origin:'Ubuntu' label:'Ubuntu' site:'192.168.10.7' isTrusted:True>])
2016-10-27 13:09:55,081 DEBUG pkgs that look like they should be upgraded: chromium-browser
chromium-browser-l10n
chromium-codecs-ffmpeg-extra

Setting to verificaton-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.1

---------------
unattended-upgrades (0.90ubuntu0.1) xenial-proposed; urgency=medium

  * Modify data/50unattended-upgrades.Ubuntu such that the release pocket is
    an allowed origin so that security updates with a new dependency will be
    upgraded and the new dependency will be installed. (LP: #1624641)

 -- Brian Murray <email address hidden> Tue, 11 Oct 2016 10:36:48 -0700

[Martin Pitt (pitti)]

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob)]

This is a regression. I tested latest updates and still having issues, albeit slightly different. Not sure if exactly the same bug and not sure if this is really a duplicate, as currently linked. But it should be checked out.

Bug #1638561

[xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob)]

Still present, or new issue due to the changes?

"""
$ apt-cache policy unattended-upgrades
unattended-upgrades:
  Installed: 0.92ubuntu1
"""

[Brian Murray (brian-murray)]

I've reuploaded this to the Yakkety queue for review as the patch was not properly applied to the package because the package's source format was native not quilt. The new upload does not use quilt and just modifies the file directly.

[xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob)]

I think it would be wise to run a regression analysis on all previous patches across all packages where, similarly, patches failed to apply and were not caught by dutifully diligent security-minded persons. i would suspect you have thousands of failed applications of patches across the board. a simple grep told me so just spending a couple minutes looking

[Martin Pitt (pitti)]

Hello Jarno, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

After this operation, 0 B of additional disk space will be used.
Get:1 http://192.168.10.7/ubuntu yakkety-proposed/main amd64 unattended-upgrades all 0.92ubuntu1.1 [33.4 kB]
Fetched 33.4 kB in 0s (582 kB/s)
The system does not support apt-btrfs-snapshot
Preconfiguring packages ...
(Reading database ... 297966 files and directories currently installed.)
Preparing to unpack .../unattended-upgrades_0.92ubuntu1.1_all.deb ...
Unpacking unattended-upgrades (0.92ubuntu1.1) over (0.92ubuntu1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (231-9ubuntu1) ...
Setting up unattended-upgrades (0.92ubuntu1.1) ...
Replacing config file /etc/apt/apt.conf.d/50unattended-upgrades with new version
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Processing triggers for man-db (2.7.5-1) ...
[ 11:53AM 5736 ] [ bdmurray@speedy:~/Pictures/Nexus 5x ]
 $ head -n7 /etc/apt/apt.conf.d/50unattended-upgrades
// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
// "${distro_id}:${distro_codename}-updates";
// "${distro_id}:${distro_codename}-proposed";
// "${distro_id}:${distro_codename}-backports";

verification done for xenial

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.92ubuntu1.1

---------------
unattended-upgrades (0.92ubuntu1.1) yakkety; urgency=medium

  * Modify data/50unattended-upgrades.Ubuntu such that the release pocket is
    an allowed origin so that security updates with a new dependency will be
    upgraded and the new dependency will be installed. (LP: #1624641)
  * Create logfile_dpkg if it does not exist so that the file can be read
    later, thereby preventing a FileNotFound crash. (LP: #1590321)
  * Create the directory /var/lib/apt/periodic/, if it does not exist, so that
    we don't receive a Traceback when trying to write a stampfile there.
    (LP: #1639977)

 -- Brian Murray <email address hidden> Mon, 07 Nov 2016 12:41:10 -0800

[xtsbdu3reyrbrmroezob (xtsbdu3reyrbrmroezob)]

Appears to be working ok now. thanks for the hard work brian and martin! :)