Comment 5 for bug 201786

If I use ssh-agent instead of seahorse, it works. Seahorse never even asks for a passphrase for the key. Hence, I think it's a seahorse bug.

You can use ssh-agent instead by editing /etc/X11/Xsession.d/60seahorse and adding unset SSH_AUTH_SOCK