Create environment and set up master node - custom_hostname failed according to we hit provisioning timeout

[Tatyanka]

Test setup master node for custom hostname check failed with :

ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.custom_hostname/fuelweb_test/helpers/decorators.py", line 96, in wrapper
    with args[0].env.d_env.get_admin_remote()\
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 395, in get_admin_remote
    password=password)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/node.py", line 171, in remote
    password=password, private_keys=private_keys)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 257, in __init__
    self.reconnect()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 300, in reconnect
    self.connect()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 295, in connect
    password=self.password)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
    raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on or 10.109.5.2

After environment revert we stay on anaconda run, and provisioning master node continues , so looks like we hit the provisioning timeout in this case. Sp lets set it bigger to avoid situation when we are affected by slow slaves

[Anastasia Kuznetsova (akuznetsova)]

Test "Create environment and set up master node" failed on 9.1 env snapshot 221 and blocked 4 Ceilometer TCs.

Link to failed job: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_ha.ceilometer/49/console

Create environment and set up master node ... 2016-09-04 21:36:38 - INFO decorators.py:119 --
<<<<<##############################[ setup_master ]##############################>>>>>
Create environment and set up master node

        Snapshot: empty

2016-09-04 21:36:56 - INFO environment.py:404 -- Waiting for admin node to start up
2016-09-04 21:36:56 - INFO environment.py:407 -- Proceed with installation
2016-09-04 21:57:02 - ERROR __init__.py:120 -- wait_for_provisioning raised: error(111, 'Connection refused')
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/__init__.py", line 113, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/models/environment.py", line 470, in wait_for_provisioning
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 126, in wait_pass
    return raising_predicate()
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/models/environment.py", line 470, in <lambda>
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 67, in tcp_ping_
    s.connect((str(host), int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

2016-09-04 21:57:02 - INFO nailgun_client.py:46 -- Initiate Nailgun client with url http://10.109.10.2:8000
2016-09-04 21:57:02 - ERROR __init__.py:120 -- generate_logs raised: ConnectFailure('Unable to establish connection to http://10.109.10.2:5000/v2.0/tokens',)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/__init__.py", line 113, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/models/nailgun_client.py", line 411, in generate_logs
    return self._put(url="/logs/package").json()
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_ha.****/fuelweb_test/models/nailgun_client.py", line 85, in _put
    return self.session.put(url=url, connect_retries=1, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 683, in put
    return self.request(url, 'PUT', **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 462, in request
    auth_headers = self.get_auth_headers(auth)
  File "/home/jenkins/venv-nailgun-tests-2.9/...

[Anastasia Kuznetsova (akuznetsova)]

Please, nominate this bug to 9.1 milestone.

[Timur Nurlygayanov (tnurlygayanov)]

Could you please fix the issue on MOS 9.1?

It is swarm blocker, 4 SWARM tests failed.

[Alexander Kurenyshev (akurenyshev)]

Timur, please attach failed jobs links or point us to that fails somehow.
It seems to me that this is invalid bug and we have observed another issue.

[Alexander Kurenyshev (akurenyshev)]

Moreover at the description there is a words `custom_hostname`
Attached at the comment #1 links includes another problem that needs to be investigated