Controller-1 upgrade failed with "/scratch not mounted or bad option"

[Heitor Matsui]

Brief Description
-----------------
DC system controller-1 host unlock failed after upgrade to latest version. Puppet log complains: "mount: /scratch not mounted or bad option". Controller-1 stays at failed state. Upgrade cannot complete.

Severity
-----------------
Critical

Steps to Reproduce
-----------------
Upgrade system controller. Follow the steps to unlock controller-1.

Expected Behavior
-----------------
Upgrade should complete successfully

Actual Behavior
-----------------
Controller-1 enter into failed state.

Reproducibility
-----------------
Reproducible

System Configuration
-----------------
DC, IPV6

Branch/Pull Time/Commit
-----------------
master

Last Pass
-----------------
N/A

Timestamp/Logs
-----------------
2022-02-10T00:56:51.815 Error: 2022-02-10 00:56:51 +0000 /Stage[main]/Platform::Filesystem::Scratch/Platform::Filesystem[scratch-lv]/Mount[scratch-lv]: Execution of '/usr/bin/mount -o remount /scratch' returned 32: mount: /scratch not mounted or bad option

Test Activity
-----------------
Feature Testing

Workaround
-----------------
lock/unlock controller recover the failure

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/831613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/831613
Committed: https://opendev.org/starlingx/stx-puppet/commit/6a0d2ab76c909d6a02114a6a4c50deecc3203971
Submitter: "Zuul (22348)"
Branch: master

commit 6a0d2ab76c909d6a02114a6a4c50deecc3203971
Author: Heitor Matsui <email address hidden>
Date: Wed Mar 2 19:20:09 2022 -0300

    Change order and attributes for mount-related resources

    After an upgrade, the host fails its first configuration attempt
    by Puppet with a mount error, but manages to configure and mount
    correctly on the second attempt. The failure happens when Puppet
    changes several attributes from /scratch on /etc/fstab, triggers
    a refresh and this in turn triggers a remount, but the fs is not
    mounted at that moment.

    This commit brings the exec resource used to manually mount the
    filesystem to execute before the mount resource, if they aren't
    mounted, as the mount resource refresh tries to remount devices
    that already were in the /etc/fstab. Also this commit changes
    the ensure attribute for scratch filesystem to 'present' because
    Puppet gets lost with this device status and then tries to run
    mount / umount operations incorrectly.

    Test Plan:
    PASS: verify that Puppet applies the configuration on the host
          after a host upgrade
    PASS: verify that /scratch mountpoint is mounted and configured
          correctly on /etc/fstab after a host upgrade
    PASS: verify that Puppet applies configuration successfully on
          a host after lock/unlock
    PASS: verify that Puppet applies configuration successfully on
          a host after a forced reboot

    Regression:
    PASS: verify that fresh install (bootstrap and host unlock)
          runs sucessfully

    Closes-bug: 1962784
    Change-Id: If2a491d3a859156ceb637cb25f213d8d6450edf3
    Signed-off-by: Heitor Matsui <email address hidden>