ssh failed to run that caused master node unaccessability by 22 port

[Tatyanka]

https://product-ci.infra.mirantis.net/view/9.0-liberty/job/9.0-liberty.ubuntu.smoke_neutron/108/console

Test failed with:
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/helpers/decorators.py", line 83, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/tests/base_test_case.py", line 167, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 444, in setup_environment
    self.wait_for_provisioning()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 519, in wait_for_provisioning
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 108, in _wait
    return raising_predicate()
  File "/home/jenkins/workspace/9.0-liberty.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 519, in <lambda>
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 56, in _tcp_ping
    s.connect((str(host), int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

after connect to vm over vncviewer I see that there is no ssh service running, after it restart node became accessible over ssh

Looks like we do not start it at all due to errr in bootsrap_master.log:
http://paste.openstack.org/show/485382/

[Tatyanka (tatyana-leontovich)]

looks like caused https://github.com/openstack/fuel-menu/commit/bf7baeb2be4cbb5bfe7b2ac0ecc168b9793438ac

[Sergey Kulanov (skulanov)]

yep, need to revert

[root@fuel ~]# fuelmenu

Traceback (most recent call last):
  File "/usr/bin/fuelmenu", line 9, in <module>
    load_entry_point('fuelmenu==9.0.0', 'console_scripts', 'fuelmenu')()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 458, in main
    setup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 314, in setup
    FuelSetup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 93, in __init__
    self.main()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 254, in main
    self.mainloop.run()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 271, in run
    self.screen.run_wrapper(self._run)
  File "/usr/lib64/python2.7/site-packages/urwid/raw_display.py", line 241, in run_wrapper
    return fn()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 336, in _run
    self.event_loop.run()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 707, in run
    self._loop()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 786, in _loop
    self._watch_files[fd]()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 387, in _update
    self.process_input(keys)
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 487, in process_input
    k = self._topmost_widget.keypress(self.screen_size, k)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 1102, in keypress
    return self.body.keypress( (maxcol, remaining), key )
  File "/usr/lib64/python2.7/site-packages/urwid/listbox.py", line 986, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 2240, in keypress
    key = w.keypress((mc,) + size[1:], key)
  File "/usr/lib64/python2.7/site-packages/urwid/container.py", line 1559, in keypress
    key = self.focus.keypress(tsize, key)
  File "/usr/lib64/python2.7/site-packages/urwid/decoration.py", line 384, in keypress
    return self._original_widget.keypress((maxcol, self.height), key)
  File "/usr/lib64/python2.7/site-packages/urwid/decoration.py", line 836, in keypress
    return self._original_widget.keypress((maxcol,maxrow-top-bottom), key)
  File "/usr/lib64/python2.7/site-packages/urwid/listbox.py", line 986, in keypress
    key = focus_widget.keypress((maxcol,),key)
  File "/usr/lib64/python2.7/site-packages/urwid/wimp.py", line 534, in keypress
    self._emit('click')
  File "/usr/lib64/python2.7/site-packages/urwid/widget.py", line 463, in _emit
    signals.emit_signal(self, name, self, *args)
  File "/usr/lib64/python2.7/site-packages/urwid/signals.py", line 120, in emit
    result |= bool(callback(*args_copy))
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/saveandquit.py", line 50, in save_and_quit
    if self.save():
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/saveandquit.py", line 56, in save
    results, modulename = self.parent.global_save()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 299, in global_save
    if module.apply(None):
  File "/usr/lib/python2.7/site-packages/fuelmenu/modules/fueluser....

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/273976

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Kulanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/273976
Reason: fixed by https://review.openstack.org/#/c/273978/

[Sergey Kulanov (skulanov)]

fixed in https://review.openstack.org/#/c/273978/

[Alexandr Kostrikov (akostrikov-mirantis)]

Seems, it is reproduced on ci[0]:
cat /var/log/fuelmenu.log
2016-03-15 21:12:24,482 DEBUG Executing command: docker ps -a
2016-03-15 21:12:24,532 DEBUG Command executed with exit code: 1
2016-03-15 21:12:24,536 DEBUG Executing command: dhcpcheck discover --timeout 5 -f json --ifaces enp0s3
2016-03-15 21:12:35,897 DEBUG Command executed with exit code: 0
2016-03-15 21:12:35,898 DEBUG No DHCP servers found
2016-03-15 21:12:35,898 DEBUG Executing command: arping -D -c3 -w1 -I enp0s3 -s 0.0.0.0 10.109.5.2
2016-03-15 21:12:37,916 DEBUG Command executed with exit code: 0
2016-03-15 21:12:37,944 ERROR Unable to read YAML: /etc/fuel/astute.yaml

I assume that by lines in third comment: "IOError: [Errno 2] No such file or directory: '/etc/fuel/astute.yaml'"

[0] https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_public/47/console

[Alexandr Kostrikov (akostrikov-mirantis)]

It has not been reproduced. Setting to 'fix released'

[Vadim Rovachev (vrovachev)]

Reproduced on 8.0.
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.fuel_master_migrate/15/console

[Sergey Kulanov (skulanov)]

@Vadim,

I'd rather vote for creating another bug with "migration flow" cause this bug should not be reproduced during provisioning/deployment master node

Also could you please also add logs from the reverted environment, why do you think it's related to fuel-menu?

[Vadim Rovachev (vrovachev)]

@Sergey,

sorry for my wrong 0 comment.

In test result we see that test fails on check migrate.
https://patching-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.fuel_master_migrate/30/testReport/(root)/Case_FuelMasterMigrate__Config_1ctrl_1comp_neutronTUN/

Log from master node:
[root@nailgun ~]# cat /var/log/fuelmenu.log
2016-07-18 11:58:12,222 DEBUG No DHCP servers found
2016-07-18 11:58:14,246 ERROR Unable to read YAML: /etc/fuel/astute.yaml
2016-07-18 11:58:14,263 ERROR Unable to read YAML: /etc/fuel/astute.yaml