Permission denied (publickey) whereas the public key has been inserted into ~/.ssh/authorized_keys: "usePAM no" issue

[jean-christophe manciot]

Security vulnerability because I cannot use SSH to connect to my Ubuntu host from a Ubuntu guest. Is Telnet the last option?
However, I can connect through the same port at the same IP address from a Windows 10 guest using the latest WinSCP software.

OpenSSH Server: 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
with openssh-server 6.9p1-2

OpenSSH client: 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
with openssh-client 6.9p1-2

Trace of the failed SSH connection:
----------------------------------------------------
root@stack:~/.ssh# ssh -v -p xxxx root@172.19.100.1
OpenSSH_6.9p1 Ubuntu-2, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.19.100.1 [172.19.100.1] port xxxx.
debug1: Connection established.
debug1: key_load_private_type: No such file or directory
debug1: key_load_private_cert: No such file or directory
debug1: key_load_private_cert: No such file or directory
debug1: key_load_private_cert: No such file or directory
debug1: key_load_private_cert: No such file or directory
debug1: key_load_private_type: No such file or directory
debug1: key_load_private_type: No such file or directory
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type 4
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9p1 Ubuntu-2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.9p1 Ubuntu-2
debug1: match: OpenSSH_6.9p1 Ubuntu-2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.19.100.1:xxxx as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes192-ctr <email address hidden> none
debug1: kex: client->server aes192-ctr <email address hidden> none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<8192<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-ed25519 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
debug1: checking without port identifier
The authenticity of host '[172.19.100.1]:xxxx ([172.19.100.1]:xxxx)' can't be established.
ED25519 key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[172.19.100.1]:xxxx' (ED25519) to the list of known hosts.
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
*************************************************************************
                            NOTICE TO USERS

This computer system is the private property of its owner, whether
individual, corporate or government. It is for authorized use only.
Users (authorized or unauthorized) have no explicit or implicit
expectation of privacy.

Any or all uses of this system and all files on this system may be
intercepted, monitored, recorded, copied, audited, inspected, and
disclosed to your employer, to authorized site, government, and law
enforcement personnel, as well as authorized officials of government
agencies, both domestic and foreign.

By using this system, the user consents to such interception, monitoring,
recording, copying, auditing, inspection, and disclosure at the
discretion of such personnel or officials. Unauthorized or improper use
of this system may result in civil and criminal penalties and
administrative or disciplinary action, as appropriate. By continuing to
use this system you indicate your awareness of and consent to these terms
and conditions of use. LOG OFF IMMEDIATELY if you do not agree to the
conditions stated in this warning.
*************************************************************************
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Offering ED25519 public key: /root/.ssh/id_ed25519
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

However; on the guest:
----------------------------------
root@stack:~/.ssh# ls -al
total 40
drwx------ 2 root root 4096 Dec 2 23:58 .
drwx------ 20 root root 4096 Nov 22 19:47 ..
-rw------- 1 root root 2 Dec 2 23:58 authorized_keys
-rw------- 1 root root 464 Dec 2 23:39 id_ed25519
-rw-r--r-- 1 root root 99 Dec 2 23:39 id_ed25519.pub
-rw------- 1 root root 1766 Dec 2 23:32 id_rsa
-rw-r--r-- 1 root root 399 Dec 2 23:32 id_rsa.pub
-rw-r--r-- 1 root root 142 Dec 2 23:59 known_hosts

On the server:
---------------------
root@msi-ge60-ubuntu:/etc/ssh# ls -al
total 308
drwx------ 2 root root 4096 Dec 3 00:22 .
drwxr-xr-x 192 root root 12288 Dec 2 23:24 ..
-rw------- 1 root root 1251 Jan 24 2015 banner-warning.txt
-rw------- 1 root root 263002 Sep 11 11:33 moduli
-rw------- 1 root root 2448 Dec 2 23:53 ssh_config
-rw------- 1 root root 3554 Dec 2 18:17 sshd_config
-rw------- 1 root root 411 Dec 2 18:21 ssh_host_ed25519_key
-rw-r--r-- 1 root root 102 Dec 2 18:21 ssh_host_ed25519_key.pub
-rw------- 1 root root 1675 Nov 25 17:00 ssh_host_rsa_key
-rw------- 1 root root 402 Nov 25 17:00 ssh_host_rsa_key.pub
-rw------- 1 root root 338 Oct 24 22:37 ssh_import_id
-rw------- 1 root root 0 Nov 25 19:41 ssh_known_hosts

and regarding the authorized keys for the root account:
root@msi-ge60-ubuntu:~/.ssh# cat authorized_keys
ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx KVM-Windows-10

ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx root@KVM-DevStack

ssh-ed25519 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx root@KVM-DevStack

[jean-christophe manciot (manciot-jeanchristophe)]

Since I'm not using default settings on the server & the client, I'm posting them here, except for the new SSH port number (!= 22):

/etc/ssh/sshd_config on the server:
-----------------------------------------------------
# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port xxxx
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
# HostKey /etc/ssh/ssh_host_dsa_key
# HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
# Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Self-Signed Certificate
# HostCertificate /root/.ssl/myCA/certs/openssh_server_crt.pem

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 2048

# OpenSSH server logs to the AUTH facility of syslog, at the INFO level.
# If you want to record more information - such as failed login attempts
# - you should increase the logging level to VERBOSE. All the details of
# ssh login attempts will be saved in your /var/log/auth.log
# By default LogLevel INFO
LogLevel VERBOSE
SyslogFacility AUTH

# Setting a lower login grace time (time to keep pending connections alive
# while waiting for authorization) can be a good idea as it frees up
# pending connections quicker but at the expense of convenience.
LoginGraceTime 30

PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication yes
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords and password authentication
# With no, only RSA public keys authentication is allowed
#PasswordAuthentication yes
PasswordAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60 by default
# Allow two pending connections. Between the third and tenth connection
# the system will start randomly dropping connections from 30% up to 100%
# at the tenth simultaneous connection.
MaxStartups 2:30:10

Banner /etc/ssh/banner-warning.txt

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
...

[jean-christophe manciot (manciot-jeanchristophe)]

If I use default/etc/ ssh_config & /etc/ sshd_config, this issue does not happen.
I'll try to isolate the setting that causes this strange behavior.

[jean-christophe manciot (manciot-jeanchristophe)]

We have a winner: setting usePAM to no generates the issue:
/etc/ssh/sshd_config
-------------------------------
...
usePAM no

ChallengeResponseAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no
...

ssh -v root@localhost
---------------------------------
...
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Offering ED25519 public key: /root/.ssh/id_ed25519
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

However, setting usePAM to yes makes the session work:
/etc/ssh/sshd_config
-------------------------------
...
use PAM yes

ChallengeResponseAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no
...

ssh -v root@localhost
--------------------------------
...
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
Enter passphrase for key '/root/.ssh/id_rsa':
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([::1]:xxx).
debug1: channel 0: new [client-session]
debug1: Requesting <email address hidden>
debug1: Entering interactive session.
debug1: client_input_global_request: rtype <email address hidden> want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Welcome to Ubuntu 15.10 (GNU/Linux 4.2.0-19-generic x86_64)
...

[Marc Deslauriers (mdeslaur)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Cesar Herrera (chg1)]

I haven't search for the information as Marc did.
But when I try to connect it says:
Permission denied (publickey).

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. I'm following up on old bugs today.

If "UsePAM yes" (the default) fixes this issue, then I'm not sure this is a bug in Ubuntu. Is this what you intended with your last comment? Or even you still consider this to be a bug in Ubuntu, please explain why and change the bug status back to New. Thanks!

[Cesar Herrera (chg1)]

When I said this I was not able to connect by ssh. But now I can.

[Launchpad Janitor (janitor)]

[Expired for openssh (Ubuntu) because there has been no activity for 60 days.]