Impossible to connect to bootstrapped node via ssh using fuel-qa

[Alexander Zatserklyany]

Steps to reproduce
Use fuel-qa
1. Setup environment
2. Bootstrap slave nodes
3. Try to connect to node by ssh_manager

Actual results:
BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Ksenia Svechnikova (kdemina)]

swarm on fuel-10.0-190 are affected:

https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.numa_cpu_pinning/21/console

[Artem Panchenko (apanchenko-8)]

Probably related patch: https://review.openstack.org/#/c/281776/

After I executed

export ENV_SLAVE_LOGIN=root
export ENV_SLAVE_PASSWORD=r00tme

SSH became to work fine for bootstrapped nodes. But need to investigate why does public key authentication fail?

[Artem Panchenko (apanchenko-8)]

Authentication using public key fails due to incorrect user name (fuel instead of root) for nodes in discovered state.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/314656

[Alexander Zatserklyany (zatserklyany)]

Bug fix proposed in https://review.openstack.org/#/c/306432
SSHManager.update_connection: add possibility to open new connection with
parameters, if no connection presents.
fuelweb_test/helpers/ssh_manager.py

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artem Panchenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/314656
Reason: closed in favor of https://review.openstack.org/#/c/306432/

[Alexander Gordeev (a-gordeev)]

It seems that fix was merged.

however, this job is still failing with the same error https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/134

BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])

[Artem Hrechanychenko (agrechanichenko)]

Fix was abandoned so wait for fix

[Alexander Zatserklyany (zatserklyany)]

Verified on fuel-9.0-312-2016-05-12_02-00-00.iso

[Artem Hrechanychenko (agrechanichenko)]

https://custom-ci.infra.mirantis.net/job/10.0.custom.packages_test.ubuntu/36/consoleFull

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/gates_tests/tests/test_review_in_fuel_agent.py", line 113, in gate_patch_fuel_agent
    verify_bootstrap_on_node(_ip, os_type="ubuntu", uuid=uuid)
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/fuelweb_test/helpers/checkers.py", line 733, in verify_bootstrap_on_node
    output = ssh_manager.execute_on_remote(ip, cmd)['stdout_str'].lower()
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/fuelweb_test/helpers/ssh_manager.py", line 193, in execute_on_remote
    result = self.execute(ip=ip, port=port, cmd=cmd)
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/fuelweb_test/helpers/ssh_manager.py", line 164, in execute
    remote = self.get_remote(ip=ip, port=port)
  File "/home/jenkins/workspace/10.0.custom.packages_test.ubuntu/fuel-qa/fuelweb_test/helpers/ssh_manager.py", line 120, in get_remote
    private_keys=keys
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 259, in __init__
    self.reconnect()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 302, in reconnect
    self.connect()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 297, in connect
    password=self.password)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 367, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 584, in _auth
    raise saved_exception
BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])

last commit in fuel-qa dir on Jenkis slave - commit 646510cab1f5cc76c816630a9c947216a882aa6e

last commit in fuel-qa - https://github.com/openstack/fuel-qa/commit/646510cab1f5cc76c816630a9c947216a882aa6e

[Dennis Dmitriev (ddmitriev)]

@Artem, it is a different issue. Patch for this bug was merged a week ago, and it works without issues since that.

The job that you linked was passed yesterday: https://custom-ci.infra.mirantis.net/job/10.0.custom.packages_test.ubuntu/35/

Please file an another bug about the issue with Ubuntu bootstrap, it should be investigated separatelly.

[Artem Panchenko (apanchenko-8)]

@Dennis,

the job you mentioned was executed with another test group and that test doesn't go to bootstrapped slave nodes via SSH. Patch that was merged a week a ago just added a possibility to update SSH connections and set credentials manually, it didn't fix the bug for existing tests.

[Dennis Dmitriev (ddmitriev)]

There is should be considered one of two ways:

1. bootstrapped nodes *should* have 'fuel/fuel' account with sudo and get rid from the 'root' access, as was planned (I think it is the most correct way);
2. all the tests that require access to the bootstrapped nodes with the deprecated way using 'root' account, should update credentials inside the test cases;
3. 'root' access to slaves should be added to the standard access ways to the slaves, and we should find a way how to detect the state of the slave before choosing the right access credentials (less preferable way, it is already out of scope of the SSHManager functionality and will introduce unnecessary logic to the system tests)

[Artem Panchenko (apanchenko-8)]

Dennis I agree with, here are my thoughts:

1. according to the specification [0] non-root users are added only on provisioned nodes, bootstrapped nodes remain with root
2. since it's node deprecated, this way sounds a bit strange, because instead of setting custom credentials for deployed environments we will manually set default 'root' user (always the same) for all connections
3. despite the fact that it's really not preferred, we already preform different actions to guess a type of the connection target [1], so IMHO adding of fallback mode [2] for slaves with 'root' user is an option too ;)

[0] http://specs.openstack.org/openstack/fuel-specs/specs/9.0/fuel-nonroot-openstack-nodes.html
[1] https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/helpers/ssh_manager.py#L108-L124
[2] https://review.openstack.org/#/c/314656/

[Alexander Gordeev (a-gordeev)]

is https://bugs.launchpad.net/fuel/+bug/1581110 duplicate too?

[Dennis Dmitriev (ddmitriev)]

@Alexander: yes, #1581110 is a duplicate.

@Artem: ok, let's use fallback mode until we find a better solution https://review.openstack.org/#/c/314656/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314656
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=d42ea1f0358cab4aec7da51f580a64df1fa3013d
Submitter: Jenkins
Branch: master

commit d42ea1f0358cab4aec7da51f580a64df1fa3013d
Author: Artem Panchenko <email address hidden>
Date: Tue May 10 18:16:39 2016 +0300

    Fallback to 'root' if provided login doesn't work

    In SSH manager try to use 'root' while connecting to
    slave nodes if authentication fails with login provided
    in settings.

    Change-Id: I647add5c67c51c26f710a3a0e92b0498478a90f9
    Closes-bug: #1580199

[Alexander Zatserklyany (zatserklyany)]

Verified on fuel-10.0-296-2016-05-31_17-01-00.iso
Fix released

[Dmitriy Kruglov (dkruglov)]

The issues is constantly reproduced on 10.0-mitaka swarm runs, 'bvt_ubuntu_bootstrap' group tests (although the latest patch seems to be in place)
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bvt_ubuntu_bootstrap/75/testReport/

[Alexander Gordeev (a-gordeev)]

the same is here https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/426/console https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/425/console

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/344785

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344785
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=56cced4df7e36d07e1aae4d0ab3b1d9bab42f323
Submitter: Jenkins
Branch: master

commit 56cced4df7e36d07e1aae4d0ab3b1d9bab42f323
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Jul 20 15:04:27 2016 +0300

    Change exception type in get_remote methon

    Workaround for non-root feature should be updated with new SSHClient
    logic. AuthenticationException is root for all unauthorized exceptions

    Change-Id: I7c973ed069393c13b01bc0f9224eef3671afd718
    Closes-bug:1580199

[Dmitry Teselkin (teselkin-d)]

Occured again https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/433/console

[Mikhail Zhnichkov (mzhnichkov)]

Nothing changed: https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/431/console

2016-07-20 13:41:42 - ERROR - ssh_client.py:141 -- Connection using stored authentication info failed!
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 120, in connect
    client.connect(**kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 603, in _auth
    raise saved_exception
BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])
2016-07-20 13:41:45 - ERROR - ssh_client.py:141 -- Connection using stored authentication info failed!
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 120, in connect
    client.connect(**kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 603, in _auth
    raise saved_exception
BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])
2016-07-20 13:41:48 - ERROR - ssh_client.py:141 -- Connection using stored authentication info failed!
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 120, in connect
    client.connect(**kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 603, in _auth
    raise saved_exception
BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])

[Alexander Gordeev (a-gordeev)]

@Dmitry, @Mikhail the referentce to the latest somehow stable fuel-qa hasn't been updated yet. It point to https://ci.fuel-infra.org/job/devops.master.env/lastSuccessfulBuild/artifact/fuel_qa_commit.txt 488c60b1a93c140cd0ba8d6d02b43d1e2ae08b8e which is not the latest.

We'll need to wait for BVT results.

[Artem Hrechanychenko (agrechanichenko)]

also need update fuel-qa repo on CI slaves for this job - review_in_fuel_agent

[Alexander Zatserklyany (zatserklyany)]

Verified:
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 119
cat /etc/fuel_build_number:
 119
cat /etc/fuel_release:
 10.0
cat /etc/fuel_openstack_version:
 newton-10.0

Fix released