ssh-askpass(-gnome) fails for ssh-add -c: agent refused operation

[schamane]

Ubuntu uses ssh-agent from OpenSSH which supports adding keys by means of `ssh-add -c` indicating that keys "should be subject to confirmation before being used for authentication. In Ubuntu 18.10 this fails with the error

  sign_and_send_pubkey: signing failed: agent refused operation

To reproduce I used a Ubuntu 18.10 Live "CD", apt-get update && apt-get upgrade, log out and log back in (these steps are not required but we want to use an up-to-date system). Then:

$ sudo apt-get install ssh-askpass-gnome
(...)
$ # verify that ssh-askpass shows a popup, confirm with Enter
$ ssh-askpass ; echo $?

0
$ ssh-keygen
(...)
$ ssh-add -D
All identities removed.
$ ssh-copy-id $sshuser@$sshserver
(...)
Number of key(s) added: 1
(...)
$ ssh $sshuser@$sshserver uname -a
Linux server 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
$ ssh-add -d
Identity removed: /home/ubuntu/.ssh/id_rsa (ubuntu@ubuntu)
$ ssh-add -c
Enter passphrase for /home/ubuntu/.ssh/id_rsa (will confirm each use):
Identity added: /home/ubuntu/.ssh/id_rsa (/home/ubuntu/.ssh/id_rsa)
The user must confirm each use of the key
$ ssh $sshuser@$sshserver uname -a
sign_and_send_pubkey: signing failed: agent refused operation
sshuser@server's password: [^C'ed]

$ ssh-add -l
2048 SHA256:yvAFsTpkNWnlrQyCp+tWV83dIF8Je3AksM0o+Ajvyyc /home/ubuntu/.ssh/id_rsa (RSA)

So, our key is loaded, ssh-askpass is working (also confirmed with `ssh-add -c </dev/null`), but authentication fails with "sign_and_send_pubkey: signing failed: agent refused operation".

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: gnome-session-bin 3.30.0-0ubuntu4
ProcVersionSignature: Ubuntu 4.18.0-10.11-generic 4.18.12
Uname: Linux 4.18.0-10-generic x86_64
ApportVersion: 2.20.10-0ubuntu13.1
Architecture: amd64
CasperVersion: 1.399
CurrentDesktop: ubuntu:GNOME
Date: Thu Jan 17 17:14:35 2019
ExecutablePath: /usr/lib/gnome-session/gnome-session-binary
LiveMediaBuild: Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)
ProcEnviron:
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: gnome-session
UpgradeStatus: No upgrade log present (probably fresh install)

[Daniel van Vugt (vanvugt)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This particular bug has already been reported and is a duplicate of bug 1789523, so it is being marked as such. Please look at the other bug report to see if there is any missing information that you can provide, or to see if there is a workaround for the bug. Additionally, any further discussion regarding the bug should occur in the other report. Feel free to continue to report any other bugs you may find.

[schamane (schamane)]

@vanugt I fail to see how bug 1789523 could be a duplicate since I have shown that ssh-agent is running and working. It just fails to call ssh-askpass in some cases.

[schamane (schamane)]

I have now removed the "marked as a duplicate of bug 1789523".

[Daniel van Vugt (vanvugt)]

Sorry, the combination of ssh-agent and "log out and log back in" made me immediately think of bug 1789523.

[Christian Ehrhardt  (paelzer)]

Hi,
I tried the same on Bionic/Cosmic/Disco (a VM each running the same commands)
1. created an ssh key (ssh-keygen)
2. added it to authorized key (cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys)
3. added it with ssh-add (ssh-add)
4. verified it was added (ssh-add -l)
5. logged to local IP via ssh using the key (ssh ubuntu@127.0.0.1 uname -a)
   (working)
6. removed the key (ssh-add -d)
7. added the key in confirm mode (ssh-add -c)
8. logged to local IP via ssh using the key (ssh ubuntu@127.0.0.1 uname -a)
   Got the confirmation prompt

I only got the "sign_and_send_pubkey: signing failed: agent refused operation" if in the prompt that asked me if I should use the key I hit "cancel".
That seems reasonable thou.

Is the following sequence failing for you?
If so would you report the full output for that sequence of commands that you get?
Note: I used a clean VM with ssh -XY to allow the popup to come through to my host.
sudo apt-get install ssh-askpass-gnome
eval `ssh-agent -s`
ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-add
ssh-add -l
ssh ubuntu@127.0.0.1 uname -a
ssh-add -d
ssh-add -c
ssh-add -l
ssh ubuntu@127.0.0.1 uname -a
  # pupup should appear

[schamane (schamane)]

@paelzer, you put a lot of work into this. Much appreciated! Thanks a lot!

There's an important difference between your approach and mine: You start another ssh-agent within a terminal that you do not leave. So, this ssh-agent is available, it's plain OpenSSH ssh-agent, nothing seems to interfere, and everything works fine. I can confirm this.

However, an ssh-agent is already running, started by the X (or GNOME) session. And that's the ssh-agent we want to use so that the keys are available to all programs started by the window manager (e.g. a file manager accessing sftp://...).

Thanks to your input, though, and thanks to the fact that I got a new VirtualBox set up, I found that it's not just `ssh-add -c` that is failing. ssh-askpass itself (tried with the plain X11 one and the GNOME version) is not shown when it should.

So, I got a step further but, unfortunately, I still don't know where the actual bug is located. I am confused by a (new?) feature of GNOME keyring (I think) that makes locally saved SSH keys available and presents a full-screen dialog to ask for the password of the key.

There are 2 big problems with this:

(1) It works only for locally saved keys, but we want ssh-agent to receive keys via ssh-add from anywhere (e.g. a remote server via `ssh -A`).

(2) This dialog is not ssh-askpass. So, if ssh-askpass is supposed to work it does not. Hence, `ssh-add -c` and key confirmations fail, too.

I am attaching my full console session in Ubuntu 18.04.1 for reference and details.

[schamane (schamane)]

Gosh, I was mistaken in #7. As in my initial report: ssh-askpass is working, it is only failing for ssh-add -c:

tux@vbu1804:~$ ssh-add -D
All identities removed.
tux@vbu1804:~$ ssh-add -l
2048 SHA256:vHjyOAyTnsn92i47AQ8qu/oL2Y3blesvs09wUoYpDDY tux@vbu1804 (RSA)
tux@vbu1804:~$ ssh-add </dev/null
Identity added: /home/tux/.ssh/id_rsa (/home/tux/.ssh/id_rsa)
tux@vbu1804:~$ # ssh-askpass dialog was shown, passphrase entered, key added:
tux@vbu1804:~$ ssh-add -l
2048 SHA256:vHjyOAyTnsn92i47AQ8qu/oL2Y3blesvs09wUoYpDDY /home/tux/.ssh/id_rsa (RSA)
tux@vbu1804:~$ ssh $USER@127.0.0.1 loginctl --no-legend
         7 1000 tux
         1 1000 tux seat0 tty1
tux@vbu1804:~$ ssh-add -D
All identities removed.
tux@vbu1804:~$ ssh-add -l
2048 SHA256:vHjyOAyTnsn92i47AQ8qu/oL2Y3blesvs09wUoYpDDY tux@vbu1804 (RSA)
tux@vbu1804:~$ ssh-add -c </dev/null
Identity added: /home/tux/.ssh/id_rsa (/home/tux/.ssh/id_rsa)
The user must confirm each use of the key
tux@vbu1804:~$ # again, ssh-askpass dialog was shown, passphrase entered, key added:
tux@vbu1804:~$ ssh-add -l
2048 SHA256:vHjyOAyTnsn92i47AQ8qu/oL2Y3blesvs09wUoYpDDY /home/tux/.ssh/id_rsa (RSA)
tux@vbu1804:~$ # but ssh fails:
tux@vbu1804:~$ ssh $USER@127.0.0.1 loginctl --no-legend
sign_and_send_pubkey: signing failed: agent refused operation
tux@127.0.0.1's password: [^C]

tux@vbu1804:~$

[Andreas Hasenack (ahasenack)]

Could this answer be related? https://unix.stackexchange.com/a/351742/254144

Are your ssh keys of the new format, having been generated with the -o option of ssh-keygen?

[schamane (schamane)]

The bug has been reproduced with the system's own ssh-keygen and the command `ssh-keygen -N $passphrase -f ~/.ssh/id_rsa`. Also, https://unix.stackexchange.com/a/351742 is about keys that didn't work at all. The problem here is that confirmation fails, everything seems to be working.

[schamane (schamane)]

I originally reported this against gnome-session. @vanvugt changed it to openssh. I think it should be changed to gnome-keyring. I didn't see it at first but now it's clear that it is actually gnome-keyring that interferes: `SSH_AUTH_SOCK=/run/user/1000/keyring/ssh`

[schamane (schamane)]

Steps for a workaround by disabling gnome-keyring's SSH agent component, tested in Ubuntu 18.04.1:

# disable GNOME Keyring's SSH Agent, cf.
# https://askubuntu.com/a/585212/43108 + https://askubuntu.com/a/607563/43108
mkdir ~/.config/autostart
cp /etc/xdg/autostart/gnome-keyring-ssh.desktop ~/.config/autostart
echo Hidden=true >> ~/.config/autostart/gnome-keyring-ssh.desktop

# add autostart .desktop for plain SSH Agent
cat > ~/.config/autostart/ssh-agent.desktop <<EOF
[Desktop Entry]
Type=Application
Name=SSH Key Agent
Exec=/usr/bin/ssh-agent
OnlyShowIn=GNOME;Unity;MATE;
X-GNOME-Autostart-Phase=PreDisplayServer
X-GNOME-AutoRestart=false
X-GNOME-Autostart-Notify=true
EOF

After a reboot `set | egrep SSH` should return something similar to
SSH_AGENT_PID=986
SSH_AUTH_SOCK=/tmp/ssh-AaTsPiP5tTcj/agent.852

[Seth Arnold (seth-arnold)]

schamane, thanks for reporting back your success. I've seen this question asked a dozen times and never seen an answer documented. Thanks!

[Christian Ehrhardt  (paelzer)]

We checked what actually is the backend that the "ssh-add -c" is trying to reach.
First we thought that should be the ssh-agent spawned for gnome-keyring-daemon [1]

In PS that is visible as:
1 1000 4029 1 20 0 656132 15860 - SLl ? 0:24 /usr/bin/gnome-keyring-daemon --daemonize --login
0 1000 26372 4029 20 0 11304 3696 - S ? 0:00 \_ /usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh

Note: there is also another ssh-agent running:
0 1000 4047 4036 20 0 568216 14944 poll_s Sl+ tty2 0:00 \_ /usr/lib/gnome-session/gnome-session-binary --session=ubuntu
1 1000 4146 4047 20 0 11304 320 - Ss ? 0:00 \_ /usr/bin/ssh-agent /usr/bin/im-launch env GNOME_SHELL_SESSION_MODE=ubuntu gnome-session --session=ubuntu

That binary is the one of the openssh package that we have proven before to work fine.:
$ dpkg -S /usr/bin/ssh-agent
openssh-client: /usr/bin/ssh-agent

But it is interesting that it uses
  -a /run/user/1000/keyring/.ssh
But the env var is actually different (no . in the filename):
$ echo $SSH_AUTH_SOCK
  /run/user/1000/keyring/ssh

But look at the ownership of the socket that we found in the env var:

$ sudo lsof +fg /run/user/1000/keyring/ssh
COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME
gnome-key 4029 paelzer 14u unix RW,ND,0x80000 0xffff910ac4316800 0t0 130189 /run/user/1000/keyring/ssh type=STREAM

So while there is a real /usr/bin/ssh-agent running the actual socket that the env variable points to is actually owned by gnome-keyring process.

Above we have proven that with a classic ssh-agent it works fine (comment #6).
The bug task for openssh is invalid for now due to that.

Maybe the gnome-keyring backend doesn't even have the -c feature [1] doesn't list it - then it would be a feature request there.
But in any case we need to re-triage that at gnome-keyring, so that is the package I'm adding a bug task for.

[1]: https://wiki.gnome.org/Projects/GnomeKeyring/Ssh

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ulrich Schwarz (uschwarz)]

My suspicion is that we never get as far as askpass – gnome-keyring-daemon spawns a ssh-agent to pass requests on to (in my case, that's easy to spot because I use PKCS#11 tokens and that's the one spawning ssh-pkcs11-helpers), and that one doesn't have DISPLAY in its environment, so it won't even try to obtain permission that way.

I'll see if I have enough foo to either force the ssh agents into debug mode (readpass.c's read_passphrase() only logs at debug level) or frobnicate their environment to have a DISPLAY, see if that fixes things.

[Ulrich Schwarz (uschwarz)]

Issuing call setenv("DISPLAY", ":0", 1) via gdb seems to result in a dialog popping up, as desired.