Ceph client role fails when Ceph monitor listens on non-default SSH port

[Chris Martin]

Some organizations (including mine) have a security policy that hosts should listen on a non-default SSH port. This causes the ceph_client role to fail, because port 22 is hard-coded in /etc/ansible/roles/ceph_client/ceph_get_mon_host.yml:
```
# look for 1 ceph monitor host that is up
- name: Verify Ceph monitors are up
  # using netcat instead of wait_for allows to both check the rc and the
  # output, rc not being available using wait_for + failed_when: false
  # failed_when: false is needed to not loose any hosts, as this check expects
  # some to be down.
  local_action: command nc -w 1 {{ item }} 22
  with_items: "{{ ceph_mons }}"
  changed_when: false
  failed_when: false
  register: ceph_mon_upcheck
```

Instead of netcat we could use the Ansible ping module: http://docs.ansible.com/ansible/ping_module.html
Not to be confused with ICMP ping, this checks for SSH connectivity and existence of Python on the target. It connects using whatever SSH port is defined ~/.ssh/config (or ansible_port if it's defined for the host in inventory). Then, the following task -- "Set ceph_mon_host to an online monitor host" -- could just set ceph_mon_host to whichever host's ping returned "pong".

If we really want to use netcat, we could variablize the SSH port as something like {{ ceph_mon_ssh_port }}, with default of 22 that the user can override, and document this option. I don't love this solution because I've already defined my SSH port in ~/.ssh/config.

[Chris Martin (6-chris-z)]

After some testing, I don't actually think the ping module will help. If it can't reach a target via SSH then it aborts playbook execution with a "fatal"/"unreachable" error.

Instead, the following code seems to work for testing SSH reachability. It uses whatever SSH users and ports are defined in ~/.ssh/config, the same way that you override these defaults for OSA overall.

```
- name: Verify Ceph monitors are up
  local_action: shell ssh {{ item }} "echo pong"
  with_items: "{{ ceph_mons }}"
  changed_when: false
  failed_when: false
  register: ceph_mon_upcheck
  tags:
  - ceph-config-create-config
  - ceph-auth-client-keyrings
  - ceph-auth-nova-libvirt-secret

- name: Set ceph_mon_host to an online monitor host
  set_fact:
                  ceph_mon_host: '{{ item.item }}'
  when: item.stdout == 'pong'
  with_items: "{{ ceph_mon_upcheck.results }}"
  tags:
  - ceph-config-create-config
  - ceph-auth-client-keyrings
  - ceph-auth-nova-libvirt-secret
```

[Chris Martin (6-chris-z)]

Proposed fix: https://review.openstack.org/#/c/393536

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393536
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-ceph_client/commit/?id=f30ee47ed0e423a0555888b45b14999e7da517bd
Submitter: Jenkins
Branch: master

commit f30ee47ed0e423a0555888b45b14999e7da517bd
Author: Chris Martin <email address hidden>
Date: Thu Nov 3 16:27:38 2016 -0700

    Support Ceph monitors on non-default SSH port

    Test for SSH reachability of Ceph monitor using ssh command
    (instead of netcat with a hard-coded port), so that users can
    override the SSH port for Ceph monitors in the usual place
    (~/.ssh/config). Makes this role work for Ceph deployments with
    monitors that listen on a non-default SSH port.

    Change-Id: I95fa8c7b7a9f12eaa9308eb51981983f4056a6f1
    Closes-Bug: 1635880

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-ceph_client 15.0.0.0b1 development milestone.