Fuel for OpenStack

Fill root filesystem on primary controller and check pacemakerTest fails on attempts to connect to online controllers

Bug #1570943 reported by Tatyanka on 2016-04-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	High	Egor Kotko	Fuel for OpenStack 10.0
	Mitaka	Fix Released	High	Egor Kotko	Fuel for OpenStack 9.0

Bug Description

Steps to reproduce:

1. run systest Case_FillRootPrimaryController__Config_ceph_all_on_neutron_vlan

Expected:
It passed

Actual
If failed to ssh on online controllers to get pcs status:
u'online':True,
u'progress':100,
u'pending_roles':[
],
u'status':u'ready',
u'mac':u'64:fe:70:3b:07:8f',
u'manufacturer':u'QEMU',
'devops_name':u'slave-02',
u'name':u'slave-02_controller',
u'roles':[
u'controller'
with :
Errno -2] Name or service not known

Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/9.0.system_test.ubuntu.filling_root/system_test/core/factory.py", line 37, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/systemtest/helpers/decorators.py", line 40, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/systemtest/actions/strengthactions.py", line 141, in getpcsinitialstate
pcsstatus = parsepcsstatusxml(remote)
File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/fuelwebtest/helpers/pacemaker.py", line 107, in parsepcsstatusxml
remoteip, 'pcs status xml')['stdoutstr']
File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/fuelwebtest/helpers/sshmanager.py", line 165, in executeonremote
result = self.execute(ip=ip, port=port, cmd=cmd)
File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/fuelwebtest/helpers/sshmanager.py", line 141, in execute
remote = self.getremote(ip=ip, port=port)
File "/home/jenkins/workspace/9.0.systemtest.ubuntu.fillingroot/fuelwebtest/helpers/sshmanager.py", line 101, in getremote
privatekeys=keys File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 259, in init self.reconnect() File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 302, in reconnect self.connect() File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper return func(*args, **kwargs) File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 289, in connect password=self.password, pkey=privatekey)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 283, in connect
totry = list(self.familiesandaddresses(hostname, port))
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/client.py", line 187, in familiesandaddresses addrinfos = socket.getaddrinfo(hostname, port, socket.AFUNSPEC, socket.SOCK_STREAM)
gaierror: [Errno -2] Name or service not known

in system_test.log:
2016-04-15 01:14:32,585 - DEBUG __init__.py:61 -- Done: get_node_ip_by_devops_name with result: 10.109.15.4
2016-04-15 01:14:32,590 - DEBUG helpers.py:284 -- Connect to '10.109.15.2:22' as 'root:r00tme'
2016-04-15 01:14:32,603 - INFO transport.py:1545 -- Connected (version 2.0, client OpenSSH_6.6.1)
2016-04-15 01:14:32,707 - INFO transport.py:1545 -- Authentication (password) successful!
2016-04-15 01:14:32,849 - INFO sftp.py:129 -- [chan 0] Opened sftp connection (server version 3)
2016-04-15 01:14:32,849 - DEBUG sftp.py:129 -- [chan 0] lstat('/root/.ssh/id_rsa')
2016-04-15 01:14:32,857 - DEBUG helpers.py:284 -- Connect to '10.109.15.2:22' as 'root:r00tme'
2016-04-15 01:14:32,867 - INFO transport.py:1545 -- Connected (version 2.0, client OpenSSH_6.6.1)
2016-04-15 01:14:32,956 - INFO transport.py:1545 -- Authentication (password) successful!
2016-04-15 01:14:33,117 - INFO sftp.py:129 -- [chan 0] Opened sftp connection (server version 3)
2016-04-15 01:14:33,117 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/id_rsa', 'r')
2016-04-15 01:14:33,119 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/id_rsa', 'r') -> 00000000
2016-04-15 01:14:33,122 - DEBUG sftp.py:129 -- [chan 0] close(00000000)
2016-04-15 01:14:33,132 - DEBUG helpers.py:284 -- Connect to '10.109.15.2:22' as 'root:r00tme'
2016-04-15 01:14:33,142 - INFO transport.py:1545 -- Connected (version 2.0, client OpenSSH_6.6.1)
2016-04-15 01:14:33,244 - INFO transport.py:1545 -- Authentication (password) successful!
2016-04-15 01:14:33,425 - INFO sftp.py:129 -- [chan 0] Opened sftp connection (server version 3)
2016-04-15 01:14:33,425 - DEBUG sftp.py:129 -- [chan 0] lstat('/root/.ssh/bootstrap.rsa')
2016-04-15 01:14:33,434 - DEBUG helpers.py:284 -- Connect to '10.109.15.2:22' as 'root:r00tme'
2016-04-15 01:14:33,448 - INFO transport.py:1545 -- Connected (version 2.0, client OpenSSH_6.6.1)
2016-04-15 01:14:33,556 - INFO transport.py:1545 -- Authentication (password) successful!
2016-04-15 01:14:33,713 - INFO sftp.py:129 -- [chan 0] Opened sftp connection (server version 3)
2016-04-15 01:14:33,713 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/bootstrap.rsa', 'r')
2016-04-15 01:14:33,715 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/bootstrap.rsa', 'r') -> 00000000
2016-04-15 01:14:33,717 - DEBUG sftp.py:129 -- [chan 0] close(00000000)
2016-04-15 01:14:33,718 - DEBUG helpers.py:284 -- Connect to '10.109.15.4:22' as 'root:r00tme'
2016-04-15 01:14:33,737 - INFO transport.py:1545 -- Connected (version 2.0, client OpenSSH_6.6.1p1)
2016-04-15 01:14:33,964 - INFO transport.py:1545 -- Authentication (publickey) successful!
2016-04-15 01:14:34,247 - INFO sftp.py:129 -- [chan 0] Opened sftp connection (server version 3)
2016-04-15 01:14:34,247 - DEBUG __init__.py:61 -- Done: get_ssh_for_node with result: <devops.helpers.helpers.SSHClient object at 0x7f0b71198e90>
2016-04-15 01:14:34,247 - DEBUG ssh_manager.py:92 -- SSH_MANAGER:Create new connection for <devops.helpers.helpers.SSHClient object at 0x7f0b71198e90>:22
2016-04-15 01:14:34,247 - DEBUG ssh_manager.py:104 -- SSH_MANAGER:Return existed connection for 10.109.15.2:22
2016-04-15 01:14:34,247 - DEBUG ssh_manager.py:105 -- SSH_MANAGER: Connections {(u'10.109.15.4', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b7007ce50>, ('10.109.15.2', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b758210d0>, (u'10.109.15.3', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b6b755990>, (u'10.109.15.8', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b6b6ee050>, (u'10.109.15.6', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b6b755590>, (u'10.109.15.7', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b711caf90>, (u'10.109.15.5', 22): <devops.helpers.helpers.SSHClient object at 0x7f0b7126de50>}
2016-04-15 01:14:34,247 - DEBUG helpers.py:357 -- Executing command: 'cd ~'
2016-04-15 01:14:34,270 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/id_rsa', 'r')
2016-04-15 01:14:34,310 - DEBUG sftp.py:129 -- [chan 0] open('/root/.ssh/id_rsa', 'r') -> 00000000
2016-04-15 01:14:34,313 - DEBUG sftp.py:129 -- [chan 0] close(00000000)
2016-04-15 01:14:34,314 - DEBUG helpers.py:284 -- Connect to '<devops.helpers.helpers.SSHClient object at 0x7f0b71198e90>:22' as 'root:r00tme'
2016-04-15 01:14:38,299 - DEBUG helpers.py:284 -- Connect to '<devops.helpers.helpers.SSHClient object at 0x7f0b71198e90>:22' as 'root:r00tme'
2016-04-15 01:14:41,306 - DEBUG helpers.py:284 -- Connect to '<devops.helpers.helpers.SSHClient object at 0x7f0b71198e90>:22' as 'root:r00tme'
2016-04-15 01:14:41,310 - INFO sftp.py:129 -- [chan 0] sftp session closed.
2016-04-15 01:14:41,313 - DEBUG __init__.py:55 -- Calling: generate_logs with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7f0b71198c50), url:None],) {}
2016-04-15 01:14:41,315 - DEBUG http.py:116 -- Set X-Auth-Token to 8c3d915bd96a43308023e19c67b72540
2016-04-15 01:14:41,457 - DEBUG __init__.py:61 -- Done: generate_logs with result: {u'status': u'running', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 0, u'message': None, u'id': 13, u'uuid': u'f282c457-109f-4337-9e26-25cd8b8ff762'}

Tags:

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2016-04-15:

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.filling_root/78/artifact/logs/sys_test.log

Nastya Urlapova (aurlapova) on 2016-04-18

Changed in fuel:
status:	New → Confirmed

Egor Kotko (ykotko) on 2016-05-04

Changed in fuel:
assignee:	Fuel QA Team (fuel-qa) → Egor Kotko (ykotko)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-05: Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/313055

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-12: Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/313055
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=cd328ac3e09751bb6b81caf9301634f873910be7
Submitter: Jenkins
Branch: master

commit cd328ac3e09751bb6b81caf9301634f873910be7
Author: Egor Kotko <email address hidden>
Date: Thu May 5 18:46:21 2016 +0200

Change the parameter

Change giving to method (parse_pcs_status_xml)
parameter from remote to ip address

Change-Id: Idda3a5912445b4fb46dee69a2ee166284462db8f
Closes-bug: #1570943

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-12: Fix proposed to fuel-qa (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/315487

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-14: Fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/315487
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=e6099585074f6f55defbdac302f35fd6c260ad92
Submitter: Jenkins
Branch: stable/mitaka

commit e6099585074f6f55defbdac302f35fd6c260ad92
Author: Egor Kotko <email address hidden>
Date: Thu May 5 18:46:21 2016 +0200

Change the parameter

Change giving to method (parse_pcs_status_xml)
parameter from remote to ip address

    Change-Id: Idda3a5912445b4fb46dee69a2ee166284462db8f
    Closes-bug: #1570943
    (cherry picked from commit cd328ac3e09751bb6b81caf9301634f873910be7)

Nastya Urlapova (aurlapova) on 2016-06-21

Changed in fuel:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.