OpenStack-Ansible

ceph_mon check fails

Bug #1751722 reported by PerToft on 2018-02-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack-Ansible	Invalid	Undecided	Unassigned

Bug Description

SSH agent forwardning is working from the glance containers.

I have resolved the problem by copying the ssh private key to the container, but this is no optimal.

TASK [ceph_client : Verify Ceph monitors are up] ********************************************************************************************************************************************************************************************
Monday 26 February 2018 07:55:32 +0000 (0:00:00.481) 0:03:38.552 *******
<cloudbc2bl03_glance_container-b229f13e> Because this is a task using "delegate_to" pipelining has been disabled. but will be restored upon completion of this task.
ok: [cloudbc2bl03_glance_container-b229f13e -> localhost] => (item=192.168.3.154) => {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.3.154", "ex
it"], "delta": "0:00:06.261435", "end": "2018-02-26 07:55:39.637732", "failed": false, "failed_when_result": false, "item": "192.168.3.154", "rc": 255, "start": "2018-02-26 07:55:33.376297", "stderr": "ssh: connect to host 192.168.3.154
port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.3.154 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}
ok: [cloudbc2bl03_glance_container-b229f13e -> localhost] => (item=192.168.3.174) => {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.3.174", "ex
it"], "delta": "0:00:07.259885", "end": "2018-02-26 07:55:47.408614", "failed": false, "failed_when_result": false, "item": "192.168.3.174", "rc": 255, "start": "2018-02-26 07:55:40.148729", "stderr": "ssh: connect to host 192.168.3.174
port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.3.174 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}
ok: [cloudbc2bl03_glance_container-b229f13e -> localhost] => (item=192.168.10.30) => {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.10.30", "ex
it"], "delta": "0:00:07.276161", "end": "2018-02-26 07:55:55.198674", "failed": false, "failed_when_result": false, "item": "192.168.10.30", "rc": 255, "start": "2018-02-26 07:55:47.922513", "stderr": "ssh: connect to host 192.168.10.30
port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.10.30 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}

TASK [ceph_client : Set ceph_mon_host to an online monitor host] ****************************************************************************************************************************************************************************
Monday 26 February 2018 07:55:55 +0000 (0:00:22.443) 0:04:00.996 *******
skipping: [cloudbc2bl03_glance_container-b229f13e] => (item={'_ansible_parsed': True, 'stderr_lines': [u'ssh: connect to host 192.168.3.154 port 22: Connection timed out'], u'cmd': [u'ssh', u'-o', u'ConnectTimeout=6', u'-o', u'UserKnownHostsFile=/dev/null', u'-o', u'StrictHostKeyChecking=no', u'192.168.3.154', u'exit'], u'end': u'2018-02-26 07:55:39.637732', '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, '_ansible_item_result': True, u'changed': False, u'stdout': u'', 'item': u'192.168.3.154', u'delta': u'0:00:06.261435', u'stderr': u'ssh: connect to host 192.168.3.154 port 22: Connection timed out', u'rc': 255, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.3.154 exit', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2018-02-26 07:55:33.376297', 'failed': False}) => {"changed": false, "item": {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.3.154", "exit"], "delta": "0:00:06.261435", "end": "2018-02-26 07:55:39.637732", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.3.154 exit", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}}, "item": "192.168.3.154", "rc": 255, "start": "2018-02-26 07:55:33.376297", "stderr": "ssh: connect to host 192.168.3.154 port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.3.154 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}, "skip_reason": "Conditional result was False", "skipped": true}
skipping: [cloudbc2bl03_glance_container-b229f13e] => (item={'_ansible_parsed': True, 'stderr_lines': [u'ssh: connect to host 192.168.3.174 port 22: Connection timed out'], u'cmd': [u'ssh', u'-o', u'ConnectTimeout=6', u'-o', u'UserKnownHostsFile=/dev/null', u'-o', u'StrictHostKeyChecking=no', u'192.168.3.174', u'exit'], u'end': u'2018-02-26 07:55:47.408614', '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, '_ansible_item_result': True, u'changed': False, u'stdout': u'', 'item': u'192.168.3.174', u'delta': u'0:00:07.259885', u'stderr': u'ssh: connect to host 192.168.3.174 port 22: Connection timed out', u'rc': 255, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.3.174 exit', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2018-02-26 07:55:40.148729', 'failed': False}) => {"changed": false, "item": {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.3.174", "exit"], "delta": "0:00:07.259885", "end": "2018-02-26 07:55:47.408614", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.3.174 exit", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}}, "item": "192.168.3.174", "rc": 255, "start": "2018-02-26 07:55:40.148729", "stderr": "ssh: connect to host 192.168.3.174 port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.3.174 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}, "skip_reason": "Conditional result was False", "skipped": true}
skipping: [cloudbc2bl03_glance_container-b229f13e] => (item={'_ansible_parsed': True, 'stderr_lines': [u'ssh: connect to host 192.168.10.30 port 22: Connection timed out'], u'cmd': [u'ssh', u'-o', u'ConnectTimeout=6', u'-o', u'UserKnownHostsFile=/dev/null', u'-o', u'StrictHostKeyChecking=no', u'192.168.10.30', u'exit'], u'end': u'2018-02-26 07:55:55.198674', '_ansible_no_log': False, '_ansible_delegated_vars': {'ansible_delegated_host': u'localhost', 'ansible_host': u'localhost'}, '_ansible_item_result': True, u'changed': False, u'stdout': u'', 'item': u'192.168.10.30', u'delta': u'0:00:07.276161', u'stderr': u'ssh: connect to host 192.168.10.30 port 22: Connection timed out', u'rc': 255, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.10.30 exit', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2018-02-26 07:55:47.922513', 'failed': False}) => {"changed": false, "item": {"changed": false, "cmd": ["ssh", "-o", "ConnectTimeout=6", "-o", "UserKnownHostsFile=/dev/null", "-o", "StrictHostKeyChecking=no", "192.168.10.30", "exit"], "delta": "0:00:07.276161", "end": "2018-02-26 07:55:55.198674", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "ssh -o ConnectTimeout=6 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 192.168.10.30 exit", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}}, "item": "192.168.10.30", "rc": 255, "start": "2018-02-26 07:55:47.922513", "stderr": "ssh: connect to host 192.168.10.30 port 22: Connection timed out", "stderr_lines": ["ssh: connect to host 192.168.10.30 port 22: Connection timed out"], "stdout": "", "stdout_lines": []}, "skip_reason": "Conditional result was False", "skipped": true}

TASK [ceph_client : Fail when no ceph_mon_host is reachable] ********************************************************************************************************************************************************************************
Monday 26 February 2018 07:55:56 +0000 (0:00:00.877) 0:04:01.873 *******
fatal: [cloudbc2bl03_glance_container-b229f13e]: FAILED! => {"changed": false, "failed": true, "msg": "No Ceph Monitor reachable"}

Monday 26 February 2018 07:55:57 +0000 (0:00:01.823) 0:04:03.696 *******
===============================================================================
ceph_client : Verify Ceph monitors are up ------------------------------ 22.44s
os_glance : Deploy Glance configuration files -------------------------- 16.93s
os_glance : Install requires pip packages ------------------------------- 8.76s
os_glance : Load glance metadata definitions ---------------------------- 6.97s
os_glance : Perform a Glance DB sync ------------------------------------ 6.92s
pip_install : Install UCA repository key -------------------------------- 5.80s
pip_install : Install PIP ----------------------------------------------- 5.09s
ceph_client : include --------------------------------------------------- 4.94s
Ensure rabbitmq user ---------------------------------------------------- 4.55s
os_glance : Create glance dir ------------------------------------------- 4.04s
os_glance : Ensure glance service --------------------------------------- 3.90s
Ensure Rabbitmq vhost --------------------------------------------------- 3.78s
Set haproxy service state ----------------------------------------------- 3.65s
galera_client : Install pip packages ------------------------------------ 3.64s
os_glance : Create tmpfiles.d entry ------------------------------------- 3.55s
pip_install : Install pip packages -------------------------------------- 3.55s
os_glance : Ensure glance user ------------------------------------------ 3.39s
os_glance : Ensure glance user to admin role ---------------------------- 3.36s
Grant access to the DB for the service ---------------------------------- 3.31s
os_glance : Ensure glance endpoint -------------------------------------- 3.20s

EXIT NOTICE [Playbook execution failure] **************************************
===============================================================================

Revision history for this message

Jean-Philippe Evrard (jean-philippe-evrard) wrote on 2018-03-06:

Could you tell us how are the key used?
Does the deploy node ssh key doesn't allow to connect on the ceph mon?
Or is it that the glance target node can't connect on the ceph mon?

What's your expectation here, what's the issue?

Thank you in advance.

Jean-Philippe Evrard (jean-philippe-evrard) on 2018-03-06

Changed in openstack-ansible:
assignee:	nobody → Logan V (loganv)

Revision history for this message

PerToft (per-minfejl) wrote on 2018-03-07:

A very clever user on #openstack-ansible irc pointed out that the command was executed as a local_action. hence, it was executed on the deployment host - and not on the glance containers as I expected.

It turned out that i has a routing problem with my ansible deployment host, which could not access the ceph mon hosts.

This issue may be closed. However, a helpfull error text describing that the deployment host is not able to access the ceph mons would be nice.

- name: Verify Ceph monitors are up
  local_action: >
    command ssh
    -o ConnectTimeout=6
    -o UserKnownHostsFile=/dev/null
    -o StrictHostKeyChecking=no
    {{ item }}
    exit

Jean-Philippe Evrard (jean-philippe-evrard) on 2018-03-14

Changed in openstack-ansible:
status:	New → Invalid
assignee:	Logan V (loganv) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.