Can't sftp into upgraded subcloud

[Reinildes Oliveira]

System Config
------------------------------------------------
subcloud

Description of failure
------------------------------------------------

Subcloud has been successfully upgraded from stx 6 to 7 but can't sftp into the subcloud from any source (System Controller or utility servers) after the upgrade given the sftp entry points to an invalid path

Impact of Failure
------------------------------------------------
Minor

Impact on users
------------------------------------------------
Can't sftp into the subcloud from any source (System Controller or utility servers) after the upgrade

Time-line based on log analysis
------------------------------------------------
1. subcloud has been successfully upgraded
2. Can't sftp into the subcloud from any source (System Controller or utility servers) after the upgrade
3. the sshd configuration file seems to be the culprit. The sftp subsystem entry points to an invalid path for the sftp-server binary

root@controller-0:/etc# grep sftp ssh/sshd_config
Subsystem sftp /usr/libexec/openssh/sftp-server
root@controller-0:/etc# ls -l /usr/libexec/openssh/sftp-server
ls: cannot access '/usr/libexec/openssh/sftp-server': No such file or directory
root@controller-0:/etc#

4. same line on the system controller points to an existing binary and sftp works as expected there

[sysadmin@controller-0 ~(keystone_admin)]$ grep sftp /etc/ssh/sshd_config
Subsystem sftp /usr/lib/openssh/sftp-server
[sysadmin@controller-0 ~(keystone_admin)]$ ls -l /usr/lib/openssh/sftp-server
-rwxr-xr-x 3 root root 125056 Jan 1 1970 /usr/lib/openssh/sftp-server

5. Also interesting to note that the access attributes of the sshd_config file are different between the System Controller and the subcloud. I'm not sure which version is correct, but they should be the same.

--->subcloud: only root can read:

sysadmin@controller-0:~$ ls -la /etc/ssh/sshd_config
-rw------- 1 root root 4858 Feb 9 18:52 /etc/ssh/sshd_config

--->system controller: anyone can read:

[sysadmin@controller-0 ~(keystone_admin)]$ ls -la /etc/ssh/sshd_config
-rw-r--r-- 1 root root 4855 Feb 7 01:21 /etc/ssh/sshd_config

bash.log

2024-02-12T20:12:27.913 controller-0 -bash: info HISTORY: PID=1944665 UID=0 grep sftp ssh/sshd_config
2024-02-12T20:12:34.787 controller-0 -bash: info HISTORY: PID=1944665 UID=0 ls -la /usr/libexec/openssh/sftp-server
2024-02-12T20:14:13.928 controller-0 -bash: info HISTORY: PID=1944665 UID=0 grep sftp ssh/sshd_config
2024-02-12T20:14:18.971 controller-0 -bash: info HISTORY: PID=1944665 UID=0 ls -l /usr/libexec/openssh/sftp-server
2024-02-12T20:15:08.634 controller-0 -bash: info HISTORY: PID=1944665 UID=0 ls ls -l /usr/lib/openssh/sftp-server
2024-02-12T20:15:11.035 controller-0 -bash: info HISTORY: PID=1944665 UID=0 ls -l /usr/lib/openssh/sftp-server

auth.log

2024-02-12T20:18:45.644 controller-0 sudo: notice pam_sss(sudo:account): Access denied for user sysadmin: 10 (User not known to the underlying authentication module)
2024-02-12T20:18:45.648 controller-0 sudo: notice sysadmin : TTY=pts/2 ; PWD=/var/home/sysadmin ; USER=root ; COMMAND=/usr/bin/cat /etc/ssh/sshd_config

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/910345
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/37543ef843d667c0524b858b4359ef6498a20c49
Submitter: "Zuul (22348)"
Branch: master

commit 37543ef843d667c0524b858b4359ef6498a20c49
Author: Rei Oliveira <email address hidden>
Date: Tue Feb 27 11:42:26 2024 -0300

    Replace sftp server bin with debian's path

    The aio-sx restore playbook will restore the /etc/sshd dir
    from the centOS backup when upgrading from centos to debian.
    This will result in etc/ssh/sshd_config pointing to a sftpserver
    path that does not exist in debian. This is reproducible
    on AIO-SX standalone and subclouds, but not AIO-DX, since
    SX and DX have different upgrade paths.

    /usr/libexec/openssh/sftp-server in centos.
    /usr/lib/openssh/sftp-server in debian.

    This is a simple change that simply replaces the wrong path
    with the valid path for debian.

    Test case:

    PASS: Run AIO-SX upgrade then verify that 'sftp controller-0'
          goes thru and the connection is established.
          'Get a file' and verify the download is successful.

    Closes-Bug: 2055324
    Change-Id: I15a47dcb0ea3dae49148b4b38f722923409b41ed
    Signed-off-by: Rei Oliveira <email address hidden>