apt-key works fine, yet apt fails with "Could not execute 'apt-key'"

[pjd]

Apt can fail to verify a Release file which verifies just fine when calling apt-key directly.

Please advise how i can supply further debug information to help fix the underlying bug.

Expected:
apt-get should only report that a repository is not signed when no such signature was found.
If a signature was in fact successfully acquired but not verified, apt-get should report failure to verify instead.
apt-get should have a meaningful error message when calling apt-key fails.

Bonus:
Calling apt-key should not fail when the same thing works fine on command line.
A reference to "Debug::Acquire::gpgv" should be in apt-secure(8) documentation.

Observed:

# uname -a
Linux hostname 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:34:49 UTC 2016 i686 i686 i686 GNU/Linux
# chroot reproducable
$ uname -a
Linux hostname 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:34:49 UTC 2016 armv7l armv7l armv7l GNU/Linux

$ lsb_release -a 2>/dev/null
Distributor ID: Ubuntu
Description: Ubuntu 16.04 LTS
Release: 16.04
Codename: xenial

$ apt-get -o "Debug::Acquire::gpgv=true" update
Get:1 http://ports.ubuntu.com xenial-security InRelease [92.2 kB]
0% [1 InRelease gpgv 92.2 kB]igners
Preparing to exec: /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.jYGUCG /tmp/apt.data.uTkX1c
gpgv exited with status 111
Summary:
  Good:
  Bad:
  Worthless:
  SoonWorthless:
  NoPubKey:
Ign:1 http://ports.ubuntu.com xenial-security InRelease
Fetched 92.2 kB in 1s (79.5 kB/s)
Reading package lists... Done
W: GPG error: http://ports.ubuntu.com xenial-security InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: The repository 'http://ports.ubuntu.com xenial-security InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

$ /usr/bin/apt-key --quiet --readonly verify --status-fd /dev/stderr /tmp/apt.sig.jYGUCG /tmp/apt.data.uTkX1c
gpgv: Signature made Tue May 3 19:02:17 2016 UTC using DSA key ID 437D05B5
[GNUPG:] SIG_ID e53PXRjA/EMb7CuZJtAicvvUm60 2016-05-03 1462302137
[GNUPG:] GOODSIG 40976EAF437D05B5 Ubuntu Archive Automatic Signing Key <email address hidden>
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key <email address hidden>"
[GNUPG:] VALIDSIG 630239CC130E1A7FD81A27B140976EAF437D05B5 2016-05-03 1462302137 0 4 0 17 10 01 630239CC130E1A7FD81A27B140976EAF437D05B5
gpgv: Signature made Tue May 3 19:02:17 2016 UTC using RSA key ID C0B21F32
[GNUPG:] SIG_ID kCsrLo9VUm7YcYhhqQUw2fbWoY4 2016-05-03 1462302137
[GNUPG:] GOODSIG 3B4FE6ACC0B21F32 Ubuntu Archive Automatic Signing Key (2012) <email address hidden>
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <email address hidden>"
[GNUPG:] VALIDSIG 790BC7277767219C42C86F933B4FE6ACC0B21F32 2016-05-03 1462302137 0 4 0 1 10 01 790BC7277767219C42C86F933B4FE6ACC0B21F32

[pjd (pjd-i)]

underlying bug was not apt-related, but could be worked around by deleting the _apt user - apt will then do exactly the same thing as I successfully tested on console.

to spare future users some pain in debugging such, I recommend
- adding getpwuid(geteuid()) to the relevant error and debug messages
- referencing the existence of Debug::Acquire::gpgv in apt-key(8) or apt-secure(8)
sorry, I am not qualified to build these into a proper patch.

[janl (janl)]

This bites me as well.

I tried symlinking /usr/bin/apt-key to /bin/true but that also fails.

Strace shows:

24419 execve("/usr/bin/apt-key", ["/usr/bin/apt-key", "--quiet", "--readonly", "verify", "--status-fd", "3", "/tmp/apt.sig.masxvS", "/tmp/apt.data.Orpf94"], [/* 25 vars */]) = -1 EPERM (Operation not permitted)
24419 write(2, "Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/no.archive.ubuntu.com_ubuntu_dists_xenial_InRelease", 113) = 113
...
24418 write(2, "Sub-process apt-key returned an error code (111)", 48) = 48
24418 exit_group(111) = ?
...
24417 write(2, "gpgv exited with status 111\n", 28) = 28

So the finger pointing at gpgv seems to be in error.

Though running it with strace might change things as well.

Apparmor is running on the machine and is enforcing (as a default setting, I did nothing specific to either enable or disable anything in it, or even initiate installation or enforcement), but nothing is logged about apt-key.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel Harvey (daniel.harvey)]

Removing the _apt user as a suggested work-around does not work for me on Ubuntu Server upgraded fro 14.04 to 16.04.

This is what I see:

root@XXX:~# apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Err:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-security InRelease
Err:3 http://us.archive.ubuntu.com/ubuntu xenial-security InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Err:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
  Could not execute 'apt-key' to verify signature (is gnupg installed?)
Fetched 95.7 kB in 0s (147 kB/s)
Reading package lists... Done
W: No sandbox user '_apt' on the system, can not drop privileges
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu xenial InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu xenial-security InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial/InRelease Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-security/InRelease Could not execute 'apt-key' to verify signature (is gnupg installed?)
W: Some index files failed to download. They have been ignored, or old ones used instead.

[janl (janl)]

Agreed removing the _apt user does not help or even seem to change anything.

The permission denied at executing apt-key from apt-get smells of app-armour interfering but I have no idea how that could happen.

[janl (janl)]

It should be noted that since this totally prevents updating of the machine this is also a security bug.

[pjd (pjd-i)]

@janl
in my case, apt was unable to use the _apt user due to misconfiguration. it did report (some) failure, which to ignore would be my choice and responsibility then - there is no way of making apt resolve the _apt user issue in a way that would lead to consistently increasing system security.

hence this is not a security bug, but rather just a request for increasing verbosity in error messages - in order to ease debugging OTHER bugs.

please verify that you did indeed have the same issue with the following steps:
1. run apt, preserving the /tmp/apt.(data|sig).+ downloads (e.g. by *temporarily* chattr+a-ing your /tmp directory)
2. run the apt-key command on the exact same files from a root command line and compare output
3. run the apt-key command on the exact same files from a _apt command line and compare output

[Bjørn Magnus Mathisen (bjornmagnus)]

I am experiencing the same problem after a 14.04 -> 16.04 update:
'apt update' fails with ".. Could not execute 'apt-key' to verify signature (is gnupg installed?)"
but chattring /tmp +a (to keep the files) and executing the same apt-key commands gotten from 'apt-get -o "Debug::Acquire::gpgv=true" update' (albeit with --status-fd pointing to /dev/stderr) works!
Right now my system is totally unable to update !

[pjd (pjd-i)]

@bjornmagnus
Have you tested my other debugging approach (making apt-get NOT change user)

1. running the apt-key commands from the sandbox ("_apt") user (switching users outside of apt-get)
$ sudo -u _apt apt-key --foo --bar baz

2. deleting the _apt user (making apt fall back to not switching users)
$ deluser _apt
$ apt-get -o "Debug::Acquire::gpgv=true" update
$ # you will want to undo this, see /var/lib/dpkg/info/apt.postinst:
$ adduser --force-badname --system --home /nonexistent --no-create-home --quiet _apt

[David Kalnischkies (donkult)]

1. Removing the _apt user is really not needed nor a good idea. Its enough to have this in a config file:
APT::Sandbox::User "root"; // remove file again after testing!
2. Symlinking /usr/bin/gpgv to /bin/true will never work as verifying signatures is more involved then just checking the exit code… there are ways to have a similar effect, but as that would be an enormous security hole I am not going to describe it here for fear of someone blindly copying it. Obviously NOT a good idea at all.

Now that we have that out of the way two "common" problems:
1. Check that /tmp has reasonable permissions. It should have 1777 and be owned by root:root.
2. ls -ld /etc/apt/trusted.gpg /etc/apt/trusted.gpg.d /etc/apt/trusted.gpg.d/*
Everything shown should be owned by root:root and everything world-readable (= the last of the three r's).

(the first is hard to detect, the second has a proper warning in newer apt versions)

[dckyio (dckyio)]

This also affects me as well, after upgrading from 14.04 to 16.04.1

Following troubleshooting methods do not help:

- removing and adding _apt user.
- file permissions on /tmp directory

[Devin Carlberg (dcarlber)]

This affected me as well, after upgrading a digital ocean VM from 14.04 to 16.04.1.

I straced the call, as well as using inotify tools to capture the temporary files created. An interesting line in the strace is:

[pid 2608] execve("/usr/bin/apt-key", ["/usr/bin/apt-key", "--quiet", "--readonly", "verify", "--status-fd", "3", "/tmp/apt.sig.CbTOUo", "/tmp/apt.data.rwmcWB"], [/* 25 vars */]) = -1 EPERM (Operation not permitted)

Can post the entire strace dump if people think it would be helpful. The apt.sig file was empty, whereas the apt.data file was 13k.

[James Stevenson (ja7es)]

What I think that might be useful is being able to get a list of open file descriptors of the process and the point of the execve is being called. I suspect that its failing because it doesn't have access to something so it get an EPERM

The only reference to execve failing in the man page is because of setuid and a file system being mounted nosuid

Unless somebody wants to read the kernel code and figure out why execve can return EPERM

[Julian Kassat (j.kassat)]

I also had this issue after upgrading 14.04 to 16.04.1 and got rid of it by rebooting. Dang.

[MarcS (marc-schmitzer)]

I finally got this fixed on my box: It seems like the linux-image-generic metapackage was removed for some reason and the box had a completely outdated kernel. I downloaded current linux-image-*-generic and linux-image-extra-*-generic from packages.ubuntu.com, installed them manually via dpkg and rebooted. apt-get update now works again.

[James Stevenson (ja7es)]

I can confirm that I had the same problem with the kernel. This was due to it being a vm in a xen environment which was booting from the host kernel which was why it was not upgraded.

After upgrading the kernel and apt to the most recent version all is working normally again.

[AnBoehm (a-andreas-2)]

I also can confirm that:
Installing the package "linux-image-generic" and rebooting fixed it.
(i had version 3.5 and afterwards version 4.4)

[Viliam Dias (viliam.dias)]

Just for reference:

I had the same problem on my digital ocean droplet (after a distro upgrade).

Problem solved after upgrade the kernel in the digital ocean administration site. More details you can see here[1].

In my case the upgrade was from 3.x to 4.x serie.

[1] https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel

[Sindarina (sindarina)]

This still seems to exist in the current release of Xenial; setting the sandbox user to root bypasses the problem, leaving it at the default means any cron job that calls 'apt-get update' breaks, because gpgv exits with error 2 (unexpected error), which leads to a failure of the 'apt-key' action being executed.

The difference, as far as I can tell thus far, seems to be in that the '_apt' user cannot read the 'pubring.gpg' file that is being created in a temporary directory, which means that gpgv cannot access it when it runs;

==
[pid 10149] stat("/etc/apt/trusted.gpg", {st_mode=S_IFREG|0644, st_size=12255, ...}) = 0
[pid 10149] faccessat(AT_FDCWD, "/etc/apt/trusted.gpg", R_OK) = 0
[pid 10149] open("/tmp/tmp.OcaWlGuT32/pubring.gpg", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)
[pid 10149] write(2, "/usr/bin/apt-key: 309: /usr/bin/"..., 41) = 41
[pid 10149] write(2, "cannot create /tmp/tmp.OcaWlGuT3"..., 64) = 64
==

This problem does not occur when root is the sandbox user, set via 'APT::Sandbox::User "root";' in '/etc/apt/apt.conf'. It's the only setting present. Disable that setting and the problem returns, while running the same thing interactively works without any issues.

I'm a bit stumped, at this point, pausing my investigation for now, but logging it here in case someone else runs into this.

The warning we're seeing looks as follows;

==
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://apt-cache.domain.example/cache/us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease: Unknown error executing apt-key
==

Using the HTTPS transport to a local cache, fresh Xenial install based on the official AMI, on AWS.

[Griffin (griffinboyce)]

I encountered this bug also, and it originated with a shell script that invoked rrdtool and exported a tar file to /tmp. Once this was done, /tmp was apparently owned by user ganglia (the rrdtool user) rather than root. Chown-ing to root:root didn't fix the issue, however. Chmod-ing /tmp to 1777 *did* fix this issue and apt-key was working normally after that.

[Chris Sterling (tophergopher)]

Confirmed that by changing the permissions on /tmp with the chmod solved this issue. My permissions were incorrect before and everything works exactly as expected now.

[Milan Hauth (milahu)]

i had the same error
while chroot-ing from arch linux into ubuntu linux.

fixed the issue with:
source /etc/environment

... to set the correct $PATH

and maybe:
echo 'APT::Sandbox::User "root";' >/etc/apt/apt.conf.d/00temp

... as suggested earlier

[Raj Basnet (rajkumarbasnet)]

# apt update
Get:1 http://deb.debian.org/debian unstable InRelease [161 kB]
Get:2 https://artifacts.elastic.co/packages/7.x/apt stable InRelease [10.4 kB]
Err:1 http://deb.debian.org/debian unstable InRelease
  Unknown error executing apt-key
Err:2 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
  Unknown error executing apt-key
Get:3 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]
Get:4 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease [110 kB]
Err:3 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease
  Unknown error executing apt-key
Get:5 http://us.archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Err:4 http://apt.postgresql.org/pub/repos/apt bionic-pgdg InRelease
  Unknown error executing apt-key
Err:6 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Unknown error executing apt-key
Err:5 http://us.archive.ubuntu.com/ubuntu bionic InRelease
  Unknown error executing apt-key
Get:8 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Err:8 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
  Unknown error executing apt-key
Get:7 https://repos.citusdata.com/community/ubuntu bionic InRelease [23.2 kB]
Err:7 https://repos.citusdata.com/community/ubuntu bionic InRelease
  Unknown error executing apt-key
Reading package lists... Done
W: GPG error: http://deb.debian.org/debian unstable InRelease: Unknown error executing apt-key
E: The repository 'http://deb.debian.org/debian unstable InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: https://artifacts.elastic.co/packages/7.x/apt stable InRelease: Unknown error executing apt-key
E: The repository 'https://artifacts.elastic.co/packages/7.x/apt stable InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: GPG error: http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease: Unknown error executing ap...