ansible is unable to remotely configure a system which still has the default password set

[Allain Legacy]

Brief Description
-----------------
Ansible is unable to remotely connect and configure a target host if that host still has the default username and password.

Severity
--------
Minor

Steps to Reproduce
------------------
Install a target host.
Run Ansible remotely against that host.

Expected Behavior
------------------
The default password should be changed automatically and the playbook should continue.

Actual Behavior
----------------
The playbook fails with the following error:

TASK [prepare-env : Fail if host is not running the right image] ***************
fatal: [yow-cgcs-wildcat-7]: FAILED! => {"changed": false, "msg": "Host 128.224.150.220 does not have the right image!."}

Changing the password manually and rerunning the playbook works as expected.

Reproducibility
---------------
100%

System Configuration
--------------------
Standard

Branch/Pull Time/Commit
-----------------------
Private load rebased on May 10.

Last Pass
---------
Unknown

Timestamp/Logs
--------------
See above for failure log:

This is my command line:
ANSIBLE_CONFIG=${MY_ANSIBLE_DIR}/bootstrap/ansible.cfg ansible-playbook ${MY_ANSIBLE_DIR}/bootstrap/bootstrap.yml --limit=yow-cgcs-wildcat-7 -i ${MY_ANSIBLE_DIR}/hosts -e "override_files_dir=${MY_ANSIBLE_DIR}"

This is my hosts file:
bootstrap:
  hosts:
    cpe-0:
      ansible_host: redacted
      ansible_port: 2223
      ansible_connection: ssh
    std-0:
      ansible_host: redacted
      ansible_port: 2226
      ansible_connection: ssh
    yow-cgcs-wildcat-7:
      ansible_host: 128.224.150.220
      ansible_port: 22
      ansible_connection: ssh
  vars:
    ansible_ssh_user: wrsroot
    ansible_ssh_pass: Li69nux*
    ansible_become_pass: Li69nux*
    ansible_become: true
    password_change_responses:
      yes/no: 'yes'
      wrsroot*: 'wrsroot'
      \(current\) UNIX password: 'wrsroot'
      (?i)New password: 'Li69nux*'
      (?i)Retype new password: 'Li69nux*'

Test Activity
-------------
Developer testing

[Ghada Khalil (gkhalil)]

Marking as release gating; related to ansible deployment feature.
Medium priority as this takes away from the zero touch aspect of ansible

[Allain Legacy (alegacy)]

I tried this again with some debug information added and noticed that the SSH command used to perform the password change was incomplete. It did not include the ssh port number that was specified in the hosts file for my particular node. Since my system is behind a NAT I provide custom SSH port number for each of my systems. Changing bootstrap/roles/prepare-env/tasks/main.yml to have this line solved the issue:

        command: "ssh -p {{ ansible_port }} {{ ansible_ssh_user }}@{{ ansible_host }}"

rather than

        command: "ssh {{ ansible_ssh_user }}@{{ ansible_host }}"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/663755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663755
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=014cff7f7ce2a0d6d5dabdc442ce260228dd47cd
Submitter: Zuul
Branch: master

commit 014cff7f7ce2a0d6d5dabdc442ce260228dd47cd
Author: Tee Ngo <email address hidden>
Date: Thu Jun 6 15:48:54 2019 -0400

    Handling of unconventional port number in playbook

    This commit supports unconventional port for ssh related tasks.
    It also eliminates a pesky warning caused by turning on pipelining.

    Closes-Bug: 1829051
    Closes-Bug: 1830258
    Change-Id: I8d1f054cdec8f2ed2a3f71b70ea10a1f7643831f
    Signed-off-by: Tee Ngo <email address hidden>