SSH failed to access fresh installed controller-0

[Peng Peng]

Brief Description
-----------------
After controller-0 installation successfully, it is not able to be accessed via SSH

Severity
--------
Critical

Steps to Reproduce
------------------
Load controller-0, then try to SSH

Expected Behaviour
------------------
login controller-0 successfully

Actual Behaviour
----------------
failed to log in

Reproducibility
---------------
Reproducible 100%
This issue was not seen with loads from 2018-11-22

System Configuration
--------------------
Seen in multiple systems:
     One node system
     Dedicated storage

Branch/Pull Time/Commit
-----------------------
master as of 2018-11-25_20-18-00

Timestamp/Logs
--------------
(2018-11-26 05:48:37) [INFO] [MainThread] Found login prompt. controller-0 installation has completed

***********************************************
Initial login and password set for controller-0
***********************************************

(2018-11-26 05:48:37) [INFO] [MainThread] Looking for: password:
(2018-11-26 05:48:37) [INFO] [MainThread] Found expected text
Opening ssh connection
(2018-11-26 05:49:37) [INFO] [MainThread] Open SSH connection to wrsroot@128.224.150.221
(2018-11-26 05:49:37) [ERROR] [MainThread] Failed to login to SSH session: wrsroot@128.224.150.221

[Ghada Khalil (gkhalil)]

Looking at the code submissions since 2018-11-22, there are some changes related to openssh:

./cgcs-root/stx/stx-integ 36dc17738d9e2ef5dab689a556d7d8f33415861e Fri Nov 23 22:09:59 2018 +0000 Gerrit Code Review Merge "Refactor patches for openssh package"
./cgcs-root/stx/stx-integ 1c8c71654ff6f20cf13bcceeb861d969fd5b53a8 Wed Nov 21 01:21:12 2018 +0000 zhipeng liu Refactor patches for openssh package

I'm not sure if these are the root-cause of the issue, but they should be investigated.

[Peng Peng (ppeng)]

Full console log attached.

We did not see this issue on load 2018-11-22_20-18-01

[Don Penney (dpenney)]

The update added a new openssh-config package for the customizations we have to the configuration files. However, it also has a %post script command that disables sshd:
https://review.openstack.org/#/c/619003/4/base/openssh-config/centos/openssh-config.spec

%{_bindir}/systemctl disable sshd.service > /dev/null 2>&1 || :

I don’t know why that would be added, as we don’t do that in the openssh package today. Maybe it was intended to be a restart to pick up the config changes, but it wouldn’t strictly be needed for the install.

The commit message says the following testing was done:
Deployment test and ping test between VMs pass
Config files check pass.

so I’m surprised this issue wasn’t seen during testing for this update.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/620205

[zhipeng liu (zhipengs)]

Removed
%{_bindir}/systemctl disable sshd.service > /dev/null 2>&1 || :
in proposed patch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620205
Committed: https://git.openstack.org/cgit/openstack/stx-integ/commit/?id=d3bf698d1b8629485893815157aa7c2b8e333bb5
Submitter: Zuul
Branch: master

commit d3bf698d1b8629485893815157aa7c2b8e333bb5
Author: zhipengl <email address hidden>
Date: Tue Nov 27 18:56:37 2018 +0800

    Fix SSH access issue introduced by change 619003

    Sshd service should not be disabled by default in config package.

    Closes-Bug: 1805157

    Change-Id: I154af9f46796feff6bb10e34aa5b5795bf92c9ed
    Signed-off-by: zhipengl <email address hidden>