[swarm] fuelmenu failed with 'IOError: [Errno 25] Inappropriate ioctl for device'

[Andrey Lavrentyev]

Detailed bug description:
fuelmenu failed with 'IOError: [Errno 25] Inappropriate ioctl for device'

Traceback: http://paste.openstack.org/show/543284/

Swarm failure: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.system_test.deploy_centos_master/9/testReport/%28root%29/Case_CentosDeployCheckRadosGW__Config_centos_master_ceph_all_on_neutron_vlan/

Steps to reproduce:
Execute 'Case_CentosDeployCheckRadosGW__Config_centos_master_ceph_all_on_neutron_vlan' test
    0. Create environment, bootstrap centos_master <<< failed here
    1. Configuration releases
    2. Bootstrap slave and make snapshot
    3. Revert bootstrapped nodes
    4. Create Fuel Environment
    5. Add nodes to environment
    6. Run network checker
    7. Deploy environment
    8. Run network checker
    9. HAProxy backend checking
    10. Check Ceph status in cluster
    11. Run health checker
    12. Check the radosgw daemon is started

Expected results:
Bootstrap is OK

Actual result:
AssertionError: bootstrap failed

Description of the environment:
9.1 snapshot #60
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8748.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-nailgun-9.0.0-1.mos8748.noarch
 fuel-migrate-9.0.0-1.mos8493.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-openstack-metadata-9.0.0-1.mos8748.noarch
 fuel-library9.0-9.0.0-1.mos8493.noarch
 nailgun-mcagents-9.0.0-1.mos753.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-misc-9.0.0-1.mos8493.noarch
 fuel-mirror-9.0.0-1.mos142.noarch
 fuel-ostf-9.0.0-1.mos938.noarch
 fuel-notify-9.0.0-1.mos8493.noarch
 fuel-utils-9.0.0-1.mos8493.noarch
 python-packetary-9.0.0-1.mos142.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos285.noarch
 rubygem-astute-9.0.0-1.mos753.noarch

Logs: https://drive.google.com/open?id=0B5HPBFb7K7gXaWpzOEhZa09wRUk

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/354108

[Dmitry Guryanov (dguryanov)]

Fuelmenu can run only on TTY in interactive mode.

When this swarm test installs fuel only from ISO - script bootstrap_admin_node.sh is run by something on master-node, maybe init prosess or something else. In this case it works fine, because STDIN is a terminal.

But when update need to be installed, fuel-qa executes bootstrap_admin_node.sh after it over ssh. It seems pseudo-terminal is not allocated in this case.

If you use ssh command and specify command on command-line, like

ssh root@master-node fuelmenu # will not work
ssh -t root@master-node fuelmenu # will work

-t option forces allocation of pseudo-terminal

[Dmitry Guryanov (dguryanov)]

So let's either don't run fuelmenu, or force ssh to allocate tty.

[Aleksei Stepanov (penguinolog)]

Allocate tty on ssh patch could be made, but will not be accessible on product CI as minimum for 2 weeks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/354602

[Vladimir Khlyunev (vkhlyunev)]

@dguryanov
bootstrap_admin_node.sh right now is the only way to apply 9.1 - https://review.openstack.org/#/c/346119/ is still in development; I will prepare the workaround for not calling fuelmenu while applying 9.1 in addition to devops fix

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/354636

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/354637

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354637
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=475754156c5f396eec5918f1f0fb13cf6d7bd485
Submitter: Jenkins
Branch: master

commit 475754156c5f396eec5918f1f0fb13cf6d7bd485
Author: Vladimir Khlyunev <email address hidden>
Date: Fri Aug 12 13:27:29 2016 +0300

    Do not show fuelmenu while applying 9.x

    The updater script is still on development so we have to use
    bootstrap_admin_node.sh. Also there is an issue in devops that
    is preventing fuelmenu appearing. Lasts, we are not able to
    quit from fuelmenu after it appears.

    Change-Id: Ifea88dde2e7ccb045172603eed182bd8db12ba79
    Related-bug:1607402

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354636
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=b3f9adfcaf4f01e7c7ac8f6552ccb45e9b17922b
Submitter: Jenkins
Branch: stable/mitaka

commit b3f9adfcaf4f01e7c7ac8f6552ccb45e9b17922b
Author: Vladimir Khlyunev <email address hidden>
Date: Fri Aug 12 13:27:29 2016 +0300

    Do not show fuelmenu while applying 9.x

    The updater script is still on development so we have to use
    bootstrap_admin_node.sh. Also there is an issue in devops that
    is preventing fuelmenu appearing. Lasts, we are not able to
    quit from fuelmenu after it appears.

    Change-Id: Ifea88dde2e7ccb045172603eed182bd8db12ba79
    Related-bug:1607402

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354602
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=642962f4e124bb491939471e90a1c1b4247d5d9f
Submitter: Jenkins
Branch: master

commit 642962f4e124bb491939471e90a1c1b4247d5d9f
Author: Alexey Stepanov <email address hidden>
Date: Fri Aug 12 12:11:37 2016 +0300

    Open PTY on channel before command execution

    Change-Id: I1dc49ad66c00d3a99a3070b0aaa708bad287e885
    Closes-bug: #1607402

[Andrey Lavrentyev (alavrentyev)]

Looks like the issue is still there.

Fresh failure on the latest Swarm: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.system_test.deploy_centos_master/36/console

Traceback (most recent call last):
  File "/usr/bin/fuelmenu", line 9, in <module>
    load_entry_point('fuelmenu==9.0.0', 'console_scripts', 'fuelmenu')()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 486, in main
    setup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 333, in setup
    FuelSetup()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 106, in __init__
    self.main()
  File "/usr/lib/python2.7/site-packages/fuelmenu/fuelmenu.py", line 257, in main
    run_delegate()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 271, in run
    self.screen.run_wrapper(self._run)
  File "/usr/lib64/python2.7/site-packages/urwid/raw_display.py", line 241, in run_wrapper
    return fn()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 311, in _run
    self.draw_screen()
  File "/usr/lib64/python2.7/site-packages/urwid/main_loop.py", line 560, in draw_screen
    self.screen_size = self.screen.get_cols_rows()
  File "/usr/lib64/python2.7/site-packages/urwid/raw_display.py", line 519, in get_cols_rows
    buf = fcntl.ioctl(0, termios.TIOCGWINSZ, ' '*4)
IOError: [Errno 25] Inappropriate ioctl for device

9.1 snapshot #169

UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-22-162322
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-22-160321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

[Aleksei Stepanov (penguinolog)]

1. Fixed upgrade procedure
2. devops 3.0.2 (released already) allow to assign vty on ssh connection.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354108
Committed: https://git.openstack.org/cgit/openstack/fuel-menu/commit/?id=03d32ae9516c435d92578ffe1496e7dd5dca9738
Submitter: Jenkins
Branch: master

commit 03d32ae9516c435d92578ffe1496e7dd5dca9738
Author: Dmitry Guryanov <email address hidden>
Date: Thu Aug 11 17:27:41 2016 +0300

    Add error message, when run not on a tty

    The library, which displays user interface, urwid,
    executes ioctl on stdin file descriptor, which
    works only on ttys. Let's report error, when we run
    not on a tty to make error investigation easier.

    Change-Id: I15e80f14f65f6467eca153dc2cd73c64751c9154
    Partial-Bug: #1607402