ssh to fuel slave nodes doesn't work after re-deploy

[Anton]

Detailed bug description:
ssh connection is not possible/key can't be uploaded to fuel slave nodes after re-deploy, due to "Permission denied (publickey).":
[root@fuel ~]# ssh node-10
Warning: Permanently added 'node-10' (ECDSA) to the list of known hosts.
Permission denied (publickey).
[root@fuel ~]# ssh-copy-id -i .ssh/id_rsa.pub root@10.20.0.9
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Warning: Permanently added '10.20.0.9' (ECDSA) to the list of known hosts.
Permission denied (publickey).

We know about bug 1489882 and WA proposed there:
echo >> /etc/fuel-bootstrap-image.conf
echo 'BOOTSTRAP_SSH_KEYS="/root/.ssh/id_rsa.pub"' >> /etc/fuel-bootstrap-image.conf

But this doesn't help in our case.

After connecting to nodes locally we see that in ".ssh/authorized_keys" file only key for <email address hidden> present.
In file "/etc/ssh/sshd_config" PasswordAuthentication set to "no"

Steps to reproduce:
1) Reset environment
2) Wait when all nodes will be back online
3) Execute deploy with "Operating System Access" -> "Authorized SSH keys" inserted or with this field empty.
4) Try to connect to slave nodes from fuel master via ssh or upload key, using https://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/maintain-environment/access-shell.html

Expected result:
SSH connection to slave node established/ssh key uploaded to slave node

Actual result:
SSH connection/uploading key not possible due to "Permission denied (publickey)."
There is PasswordAuthentication set to "no" in "/etc/ssh/sshd_config" and at the same time key from "Operating System Access" -> "Authorized SSH keys" was not copied to ".ssh/authorized_keys" on slave node

Reproducibility:
100%

Workaround:
Connect to nodes locally and modify "/etc/ssh/sshd_config" and ".ssh/authorized_keys" files

Impact:
Visit to lab needed after each re-deploy

Description of the environment:
OpenStack Release Mitaka on Ubuntu 14.04
Network Neutron with tunneling segmentation
os_odl-l2_sfc-ha scenario

[Anton (achivkun)]

Hi all.

If there is some known workaround for this problem, please kindly let us know. We have tried to create "/etc/fuel-bootstrap-image.conf" and put key there before re-deploy:
echo >> /etc/fuel-bootstrap-image.conf
echo 'BOOTSTRAP_SSH_KEYS="/root/.ssh/id_rsa.pub"' >> /etc/fuel-bootstrap-image.conf

but it doesn't help in our case.

BR/Anton.

[Anton Chevychalov (achevychalov)]

Anton, I see some misunderstanding of settings page in that case.

First of all, "Operating System Access" -> "Authorized SSH keys" has no relation to access to root account. That is all about fueladmin user (or someone else you choose as "Username"). Check the settings above "Authorized SSH keys". And PasswordAuthentication setting to "no" is expected.

Besides it seems to me, that you regenerate /root/.ssh/id_rsa on Fuel Master. Am I right? That procedure is not defined and not allowed. To recover from that issue please open support ticket.

As I was not able to repeat you trouble on 9.2 lab, so I will change state of that bug to Incomplete.

[Anton (achivkun)]

Hi Anton.

Thank you for your inputs!

Yes, you are right - /root/.ssh/id_rsa has been regenerated on Fuel master node before second deploy attempt (it was not mentioned anywhere that it must be the same between deploys). I didn't see any restriction regarding this part and supposed that each new deploy should be independent from previous (i.e. I didn't expect that previous key will be considered after new deploy attempt).

BTW, please let me know where support ticket should be created?

Thank you in advance!
BR/Anton.

[Anton (achivkun)]

BTW, will it work if I will create "fueladmin" user on jump server, generate key for it and paste public key to "Authorized SSH keys" before next re-deploy? Will I have access to slave nodes from jump server then (from "fueladmin" user)? Or there is better alternative?

[Anton (achivkun)]

Just posted question: https://answers.launchpad.net/fuel/+question/556902 hope this is correct action.

[Anton Chevychalov (achevychalov)]

There is no needs to create fueladmin on jump host. You can always invoke "ssh -i <path_to_private_key> fueladmin@<node_ip>".

You are right it was never mentioned that it works like that, but that is reality. That keys have been generated on Master node installation and mentioned in /etc/nailgun/settings.yaml. So you have to put right pub key into that file and restart Nailgun. That keys is not for customer use.

[Anton (achivkun)]

Hi Anton.

To be honest, I can't find where in "settings.yaml" old key might be mentioned.

I've attached our "settings.yaml" file.
As far as I understand next 3 parameters might be important:
PATH_TO_SSH_KEY: "/root/.ssh/id_rsa" - currently in this file we have actual key (not those, which was there after Master node installation):

[root@fuel opnfv]# ssh-keygen -y -f /root/.ssh/id_rsa
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Mymded2E3JSyV+VTR37hK//pH1UzROIQhx5r+d+WgbnFBPH2wVLk8yxMDZtHAkNsHQuWaIOcP8ra26/tLkMoZnN4nbLHzbKLdzqBYyOepBxqQ4G+T886V2vVXQGZanX65szohipkjj2moajc9H+JB9AckuxCv/cZuQIiJnQlxLp/2YCz++VA+izmwHAsiVkfSoM2+iI64hxhCR3EU1+6AdHWIDxIlhG3Zjm6VMrMKLZhLAcIxJRYKMvM8fuvSRAeZnMZN4wDPMorOdWQOQPr8QnUtqNuEnTHto12NSXsjogP9+9cef3Y/ebBfNDKBarWfkwR9OMTBlATu/R1j8bl
[root@fuel opnfv]# cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Mymded2E3JSyV+VTR37hK//pH1UzROIQhx5r+d+WgbnFBPH2wVLk8yxMDZtHAkNsHQuWaIOcP8ra26/tLkMoZnN4nbLHzbKLdzqBYyOepBxqQ4G+T886V2vVXQGZanX65szohipkjj2moajc9H+JB9AckuxCv/cZuQIiJnQlxLp/2YCz++VA+izmwHAsiVkfSoM2+iI64hxhCR3EU1+6AdHWIDxIlhG3Zjm6VMrMKLZhLAcIxJRYKMvM8fuvSRAeZnMZN4wDPMorOdWQOQPr8QnUtqNuEnTHto12NSXsjogP9+9cef3Y/ebBfNDKBarWfkwR9OMTBlATu/R1j8bl root@fuel

PATH_TO_BOOTSTRAP_SSH_KEY: "/root/.ssh/bootstrap.rsa" - this file doesn't exist in our case.

AUTHORIZED_KEYS:
  - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFUulCbREWAVvoAKrL/zDOzVBK2GwhTQhn9U8148Uwhq3wsS3Ulum0OLz3R+oa1qygzP/TuqCry8h04QHcAFLGhATQXO3zMX6VUEjtQ0GkkcJq2L5aO0Qg8wS8h7HFmPn6dl8BUideVfD6lwQkBxuyVHUrEj/VqBlDIn2zZhDqYEa/vCfeS2tYxs/81JsCsFVHNJ3yVKPruM3dx+2ZXB+6LwSLGB05nBUXrxl0EBJ15jyWboBcRnMhdlCHFb77wF75Ins1QcRUcOa49nMc9el/sKg1SSHLWJkzB/IuHI8XSa+PNIyFzniBRoaBb5Ko7xFn8qNCXKmeDPJcMtN40FX <email address hidden>"
here we have only key for "fuel.domain.tld" domain. I'm not 100% sure, but if I remember correctly, after each re-deploy, key for this domain was new (when we logged to slave nodes manually we saw different key for this domain each time)

Could you please point out which place of this file should be updated? Should we add key for "root@fuel" to section AUTHORIZED_KEYS:
  - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFUulCbREWAVvoAKrL/zDOzVBK2GwhTQhn9U8148Uwhq3wsS3Ulum0OLz3R+oa1qygzP/TuqCry8h04QHcAFLGhATQXO3zMX6VUEjtQ0GkkcJq2L5aO0Qg8wS8h7HFmPn6dl8BUideVfD6lwQkBxuyVHUrEj/VqBlDIn2zZhDqYEa/vCfeS2tYxs/81JsCsFVHNJ3yVKPruM3dx+2ZXB+6LwSLGB05nBUXrxl0EBJ15jyWboBcRnMhdlCHFb77wF75Ins1QcRUcOa49nMc9el/sKg1SSHLWJkzB/IuHI8XSa+PNIyFzniBRoaBb5Ko7xFn8qNCXKmeDPJcMtN40FX <email address hidden>"
  - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Mymded2E3JSyV+VTR37hK//pH1UzROIQhx5r+d+WgbnFBPH2wVLk8yxMDZtHAkNsHQuWaIOcP8ra26/tLkMoZnN4nbLHzbKLdzqBYyOepBxqQ4G+T886V2vVXQGZanX65szohipkjj2moajc9H+JB9AckuxCv/cZuQIiJnQlxLp/2YCz++VA+izmwHAsiVkfSoM2+iI64hxhCR3EU1+6AdHWIDxIlhG3Zjm6VMrMKLZhLAcIxJRYKMvM8fuvSRAeZnMZN4wDPMorOdWQOQPr8QnUtqNuEnTHto12NSXsjogP9+9cef3Y/ebBfNDKBarWfkwR9OMTBlATu/R1j8bl root@fuel"

Thank you in advance!

[Anton Chevychalov (achevychalov)]

You could add new key, but that is useless, because you wiped private key. That means that you have to replace AUTHORIZED_KEYS section of settings.yaml by contents of /root/.ssh/id_rsa.pub (from master node).

[Anton (achivkun)]

Hi Anton.

Ok, thank you! At next re-deploy attempt I will try to replace key in "AUTHORIZED_KEYS" section of settings.yaml file and put there public key from /root/.ssh/id_rsa.pub (from master node). Hope, it will really help.

Thanks again for your help!
BR/Anton.

[Anton Chevychalov (achevychalov)]

Do not forget to restart nailgun.

[Anton (achivkun)]

Ok, thanks for reminding! I'll execute "service nailgun restart" after update. BTW, currently key in "settings.yaml" is for "<email address hidden>" and I'm going to replace it with key for "root@fuel". Is "root@fuel" key good enough for normal operation of environment, or there can be some drawback?

[Anton Chevychalov (achevychalov)]

It does not matter.