[BVT] Connection refused while master node provisioning

[Aleksey Zvyagintsev]

https://product-ci.infra.mirantis.net/job/9.0.main.ubuntu.bvt_2/131/consoleText

Possible issue: misconfiguration into fuel-qa network

fuel-qa trying to connect => root@10.109.20.2
otherwise in VNC we can see ip addr:
eth0: 10.20.0.2

Issue not reproducible, and env not available after
$ dos.py revert-resume 9.0.main.ubuntu.bvt_2.131 error_setup_master && ssh root@10.109.20.2
since dos.py reverts env in 'provisioning state' , w\o passing correct cmdline for dracut.

version 9.0

[Aleksey Zvyagintsev (azvyagintsev)]

add log

[Tatyanka (tatyana-leontovich)]

hm according to log fuel-qa tries to ssh to 10.109.35.2
016-03-31 02:32:53,822 - DEBUG __init__.py:55 -- Calling: get_admin_node_ip with args: ([<class 'fuelweb_test.models.environment.EnvironmentModel'>(0x7fd16fc5d2d0), ip:None],) {}
2016-03-31 02:33:03,317 - DEBUG __init__.py:61 -- Done: get_admin_node_ip with result: 10.109.35.2

Also 10.20.0.2 is default ip, so looks like changes are not applying

[Maksym Strukov (unbelll)]

The way we start master-node setup: we attach iso to vm, start it, and blindfolded waiting 3 sec before pressing <Esc> key in grub menu and type kernel params to start auto-install.
So one of real causes of this bug is overload of server. In this case after 3 seconds after we sent command to start vm, vm is not show grub menu on the screen. Then vm receive and ignore kernel params and react only to <Enter> key which start setup with default parameters.

This is very rare coincidence, but we can fix this in two ways:
1. Watch that server has free resources
2. Refactor fuel-qa and start master node directly send parameters via libvirt (but in this case we will not test ISO boot menu)

As for now I can't reproduce this bug, but assume it real.

[Aleksey Zvyagintsev (azvyagintsev)]

From my POV - we should increase amount of simulates deployment in one hypervisor

[Aleksey Zvyagintsev (azvyagintsev)]

Sorry for typo,
 * decrease for sure :)

[Aleksei Stepanov (penguinolog)]

Reproduced on local PC on BVT2, when provisioning node 6.

2016-03-31 12:50:38,853 - ERROR decorators.py:125 -- Traceback (most recent call last):
  File "/home/penguinolog/work/fuel-qa/fuelweb_test/helpers/decorators.py", line 119, in wrapper
    result = func(*args, **kwargs)
  File "/home/penguinolog/work/fuel-qa/fuelweb_test/tests/test_ceph.py", line 474, in ceph_rados_gw
    self.env.d_env.nodes().slaves[:6])
  File "/home/penguinolog/work/fuel-qa/fuelweb_test/models/environment.py", line 125, in bootstrap_nodes
    self.sync_time()
  File "/home/penguinolog/work/fuel-qa/fuelweb_test/models/environment.py", line 137, in sync_time
    new_time = sync_time(self.d_env, nodes_names, skip_sync)
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/ntp.py", line 35, in sync_time
    with GroupNtpSync(env, node_names) as g_ntp:
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/ntp.py", line 79, in __init__
    ntp = Ntp.get_ntp(get_node_remote(env, node_name),
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/helpers.py", line 160, in get_node_remote
    name=node_name).interfaces[0].mac_address)
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/helpers.py", line 191, in get_slave_ip
    js = get_nodes(admin_ip)
  File "/home/penguinolog/work/fuel-devops-venv/lib/python2.7/site-packages/devops/helpers/helpers.py", line 512, in get_nodes
    nodes_response = request.urlopen(nodes_request)
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib64/python2.7/urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

[Kyrylo Galanov (kgalanov)]

Looks like the same issue: https://product-ci.infra.mirantis.net/job/9.0.main.ubuntu.bvt_2/153/