apt signature requirements prevent updates from some repositories

[Eugene Crosser]

Since xenial updated the requirements for the strength of PGP signatures of packages, packages from some repositories are no longer updated. Apt-get update reports these errors:

E: Failed to fetch http://[...]/Release No Hash entry in Release file /var/lib/apt/lists/partial/[...] which is considered strong enough for security purposes
E: Some index files failed to download. They have been ignored, or old ones used instead.

While the motivation for the change is valid, the result is a potential security problem, as the new versions of the packages that may fix recently discovered vulnerabilities are not automatically installed.

One less important but unfortunate effect is a scary message that is displayed to the user, without clear explanation that the problem needs to be addressed by the repository owner.

Related: Bug #1558331

[Julian Andres Klode (juliank)]

This is fixed in my bugfix/sha1-deprecated branch:

 https://github.com/julian-klode/apt/compare/master...bugfix/sha1-deprecated

I spent half my night working on it. David also has a more involved branch fixing this as well and further generalizing stuff, but I think we should go with a less intrusive solution for now.

I know that it's somewhat unclear that the repository owner is responsible, but I don't really feel like adding another message for that, as I want to get the messages translated now, and I'm not sure it's possible in a sensible way. It also helps people think about what repositories they use and remove unneeded ones, like Google's talkplugin if they use Chrome.

[Eugene Crosser (crosser)]

As a sidenote, talkplugin is still needed for Firefox users (I don't know why, Firefox has fully functional WebRTC now, but there you go). Last published plugin package is of April 2015.

In the long term, it would make sense to modify upstream apt to clearly distinguish between the problems that are under the user's control and those under the repo owner's control, and report it e.g. via the return code or the error message prefix. Then GUI tools could give the user better advice than the misleading "Check your internet connection" message.

[Juho-Pekka Kuitunen (78luphr0rnk2nuqimstywepozxn9kl19tqh0tx66b5dki1xxsh5mkz9gl21a5rlwf-launchpad)]

16.04 is out and this issue is about to hit the "mainstream." I ran into it when I can't install Sensu on 16.04. I get the sentiment, what I dislike is that I have no sane workaround to continue my work.

What are we stalling on right now? What direction needs to be pushed?

[Julian Andres Klode (juliank)]

I was basically ready I think, but nobody else wanted to have it , so we're not going to do it now.

It's too late for 16.04 anyway.

If it turns out to be a huge Deal, we can probably Update it in an SRU.

[Lonnie Lee Best (launchpad-startport)]

Here's how I'm dealing with it for now:
http://askubuntu.com/a/760348/256054

[Lonnie Lee Best (launchpad-startport)]

I came across this effort at debian.org :
https://wiki.debian.org/Teams/Apt/Sha1Removal

[Ads20000 (ads20000)]

Given that another bug can't be really be fully fixed until this bug is fixed ( https://bugs.launchpad.net/ubuntu/+source/gnome-software/+bug/1573408/comments/12 ) could this be updated in an SRU please Julian (I'm just a user not a developer, mind)?

[Sebastien Bacher (seb128)]

seemslike that's creating issues with gnome-software, the /var/lib/app-info/yaml/ symlinks are not refreshed when one of the repository has an error which leads to gnome-software to be empty...
we just debugged that with dholbach yesterday which got bitten by it, he had the spotify repo enabled and changed his mirror from "de" to "us", the appstream symlinks were still pointed to ".de" filenames which were not existant

[Julian Andres Klode (juliank)]

The AppStream issue will happen with every error. AppStream would have to run unconditionally, as APT keeps the old files around in case of errors.

I don't see how the spotify thing affects this, that's only a warning.

[Fabio C. Barrionuevo (luzfcb)]

Julian, please see:
https://asciinema.org/a/1xwbdc2r2qcdbby1qpucmztvb

I do not know, but these problems with apt-get are almost making me go back to 14.04

other related problems:
https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1570141

[Julian Andres Klode (juliank)]

Hi Fabio,

in your case, everything seems to be fine signature wise on your system.

Now, the Ign you see are for InRelease files, because APT falls back to Release files a few lines later.

I don't know why you don't see the docker-engine package. Are you perhaps running i386? The docker package is only provided for amd64. You should also see the heroku-toolbelt package.

Finally, your system seems a bit messed up. Please delete the files in /var/lib/apt/lists/partial, as there seems to be some permission issue.

[Julian Andres Klode (juliank)]

I'm thinking about merging my APT branch upstream and giving it some testing on users.

For appstream, it might make sense to move the hook from APT::Update::Post-Invoke-Success to APT::Update::Post-Invoke so it runs even if the update errors - APT updates its cache as well, and makes sure the lists directory is kept in a sane state.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Iain Lane (laney)]

Julian - I just tried and it seems like Post-Invoke is run when there is a W: but not an E: - is that right?

In this snippet I've added an Apt::Update::Post-Invoke { "echo 'hello' }

Reading package lists... Done
W: http://dl.google.com/linux/musicmanager/deb/dists/stable/Release.gpg: Signature by key 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 uses weak digest algorithm (SHA1)
W: http://dl.google.com/linux/talkplugin/deb/dists/stable/Release.gpg: Signature by key 4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 uses weak digest algorithm (SHA1)
W: http://ppa.launchpad.net/pitti/systemd/ubuntu/dists/trusty/Release.gpg: Signature by key FB322597BBC86D52FEE950E299B656EA8683D8A2 uses weak digest algorithm (SHA1)
E: Failed to fetch http://dl.google.com/linux/musicmanager/deb/dists/stable/Release No Hash entry in Release file /var/lib/apt/lists/dl.google.com_linux_musicmanager_deb_dists_stable_Release which is considered strong enough for security purposes
E: Failed to fetch http://dl.google.com/linux/talkplugin/deb/dists/stable/Release No Hash entry in Release file /var/lib/apt/lists/dl.google.com_linux_talkplugin_deb_dists_stable_Release which is considered strong enough for security purposes
E: Some index files failed to download. They have been ignored, or old ones used instead.
laney@raleigh>

...and then after removing the google repositories:

Hit:10 http://ppa.launchpad.net/pitti/systemd/ubuntu trusty Release
hello
Reading package lists... Done
Building dependency tree
Reading state information... Done
9 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: http://ppa.launchpad.net/pitti/systemd/ubuntu/dists/trusty/Release.gpg: Signature by key FB322597BBC86D52FEE950E299B656EA8683D8A2 uses weak digest algorithm (SHA1)
laney@raleigh>
I see <https://anonscm.debian.org/cgit/apt/apt.git/tree/apt-pkg/update.cc#n108> which looks like it's where the function is returning from (didn't trace it though), which would prevent the Post-Inboke hook from running.

[Julian Andres Klode (juliank)]

Yes that's true currently. We're looking at changing this, and probably the meaning of success as well (to not everything failed, aa that is more realistic).

[Julian Andres Klode (juliank)]

Maybe we should split this up into the AppStream related issue and the signature error itself or repurpose it for the former? The AppStream issue will be fixed soonish by invoking -Success even when some sources failed - because some sources succeeded, so we have something new to update from (and APT updates the cache anyway).

I'd like to not complicate this any further, the error message is very useful otherwise, as users do not get notified about anything otherwise (python clients just drop all warnings). Clients should not abort on an error here anyway (for the reason given above), this instance is just more common.

[Matthias Klumpp (ximion)]

With your fix, we don't need to touch AppStream at all, since it would just work. Which would mean we could close the issue in "appstream" as invalid then...

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Fix for the AppStream issue, currently in testing" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

Dropping the AppStream task, as that's more of an APT bug. With the SRU and change in yakkety, appstream data is now generated even if some updates failed.

Note that I chose not to close this bug report with those uploads, as this bug report sort of keeps track of the larger problem of these error class, and not the appstream-related aspect in particular.

With this most important problem gone, I consider marking this bug as wontfix for xenial - Long term we will have a reworked handling of untrusted repositories, but I think this is probably going to be to invasive for xenial to backport.

[Julian Andres Klode (juliank)]

As I wrote two months ago, this seems to be wontfix for xenial, at least for know. We can eventually re-evaluate that at some point, once we can grant acceptions in 1.3 or 1.4 ,but it does not seem likely; as the changes are likely very invasive.

[Julian Andres Klode (juliank)]

Hmm, it seems I made it possible to downgrade errors to warnings (untrusted to weak) in the gpgv verification, but never in the other code part.