Can't access cephfs after upgrading Starlingx

[Thiago Oliveira Miranda]

Brief Description
Can't access the cephfs after upgrading the sub cloud

Severity
Major

Steps to Reproduce
1. Install with stx.5.0
2. configure cephfs:
sudo mount -t ceph controller-0:6789:/ /mnt/mycephfs
sudo bash -c "echo 'I am here' > /mnt/mycephfs/foo"
sudo bash -c "echo 'You are there' > /mnt/mycephfs/bar"
sudo ls -al /mnt/mycephfs
cat /mnt/mycephfs/foo
cat /mnt/mycephfs/bar

Expected Behavior
After upgrade expected to re-mount and access the files.

Actual Behavior
Failing to mount due to missing keyring

Reproducibility
seen once

System Configuration
DC-4, subcloud-12

Branch/Pull Time/Commit
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="2021-12-10_21-09-00"
SRC_BUILD_ID="19"

Last Pass
stx.5.0

Timestamp/Logs
Time: 2021-12-11T23:20:41.000

After Upgrade Error msg:
sudo mkdir /mnt/mycephfs
[sysadmin@controller-0 ~(keystone_admin)]$ sudo mount -t ceph controller-0:6789:/ /mnt/mycephfs
2021-12-11 23:20:41.464 7f57edebbb40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.guest.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

Alarms
No Alarms

Test Activity
Upgrade Testing

Workaround
Tried the following workaround:
KEYRING=$(ceph auth get client.ceph-pool-kube-cephfs-data | sed -n 's/^[[:blank:]]key[[:blank:]]+=[[:blank:]](.)/\1/p')
echo $KEYRING
CEPH_USER_KEYRING=/etc/ceph/ceph.client.guest.keyring
sudo bash -c "echo $KEYRING >$CEPH_USER_KEYRING"
sudo cat $CEPH_USER_KEYRING

ceph fs dump
sudo mkdir /mnt/mycephfs
sudo mount -t ceph controller-0:6789:/ /mnt/mycephfs
sudo ls -al /mnt/mycephfs
sudo umount /mnt/mycephfs

Mount was successful but still can't see any of the files created prior to upgrade (foo, bar)
sudo ls -al /mnt/mycephfs
Password:
total 4
drwxr-xr-x 2 root root 0 Dec 11 22:22 .
drwxr-xr-x. 6 root root 4096 Dec 11 23:20 ..

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/823016

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/823016
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/f72175fe29b5947b3276b042833da2cf8a5abe3b
Submitter: "Zuul (22348)"
Branch: master

commit f72175fe29b5947b3276b042833da2cf8a5abe3b
Author: Thiago Miranda <email address hidden>
Date: Mon Dec 27 07:51:01 2021 -0500

    Fix cephfs access after upgrading

    After the system upgrade, the ceph mds data was deleted, and we could
    no longer find the files stored with CephFS.

    This procedure resets and recreates the mds so that the files appear
    again after the volume is mounted.

    Test Plan:
     - Tested Upgrading procedure from 21.05 on AIO-SX
     - Tested B&R procedure on AIO-SX

    Closes-bug: 1955823

    Signed-off-by: Thiago Miranda <email address hidden>
    Change-Id: Ic146dda35c3c71cf11c73a4f972b274ec3404d07