Script doesn't resize platform filesystem during upgrade-activate

[Heitor Matsui]

Brief Description
-----------------
During upgrade-activate, the script 17-resize-systemcontroller-filesystems.sh will try attempt to resize backup and then platform filesystems. Backup will be resized successfully but platform won't be resized because of the validations introduced by fix of https://bugs.launchpad.net/starlingx/+bug/1968201.

Severity
--------
Minor: system won't resize platform filesystem during upgrade-activate, but won't fail the upgrade and the platform filesystem can be manually resized later

Steps to Reproduce
------------------
Upgrade all hosts to load n+1
Run system upgrade-activate on System Controller deployment with enough space free to resize backup and platform (generally 20GB free on cgts-vg)
Observe backup being resized, but not platform, on /var/log/platform.log

Expected Behavior
------------------
platform controllerfs resized to 20GB
backup filesystem on both hosts resized to fit the 20GB platform filesystem

Actual Behavior
----------------
backup filesystem resized
platform filesystem not resized to 20GB

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Standard System Controller, AIO-DX System Controller

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
N/A

Timestamp/Logs
--------------
# platform.log:
2022-05-17T16:54:30.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Starting filesystems resize on DC System Controller for increased parallel subcloud deployment for from release 21.12 to 22.06 with action activate
2022-05-17T16:54:30.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Verifying if filesystems need resizing...
2022-05-17T16:54:31.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Platform filesystem needs resizing, current size is 10G, ideal size is 20G.
2022-05-17T16:54:31.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Verifying if there is enough available space to resize...
2022-05-17T16:54:34.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: [controller-0] Available cgts-vg space: 65.812G, need 20G to resize.
2022-05-17T16:54:35.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: [controller-1] Available cgts-vg space: 65.812G, need 20G to resize.
2022-05-17T16:54:35.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: LVG cgts-vg has enough space for resizing, continuing with resize operations...
2022-05-17T16:54:35.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Trying to resize host-fs backup for both controllers...
2022-05-17T16:54:37.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: [controller-0] Current backup size is 25G, new size will be 35G.
2022-05-17T16:55:04.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Successfully resized backup filesystem on controller-0.
2022-05-17T16:55:08.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: [controller-1] Current backup size is 25G, new size will be 35G.
2022-05-17T16:55:37.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Successfully resized backup filesystem on controller-1.
2022-05-17T16:55:37.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Trying to resize controllerfs platform filesystem...
2022-05-17T16:55:37.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Current platform size is 10G, new size will be 20G.
2022-05-17T16:55:40.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Retry 1 of 5, checking if platform filesystem is resized and available...
2022-05-17T16:55:43.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Current platform fs size/state: 10/None
2022-05-17T16:56:46.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Retry 2 of 5, checking if platform filesystem is resized and available...
2022-05-17T16:56:49.000 controller-1 root: info 17-resize-systemcontroller-filesystems.sh: Current platform fs size/state: 10/None

Test Activity
-------------
Feature Testing

Workaround
----------
Manually resize platform with system controllerfs-modify command

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/842215
Committed: https://opendev.org/starlingx/config/commit/5d36bedaa5f4c5a28d0fbc079cc22bc437eb562e
Submitter: "Zuul (22348)"
Branch: master

commit 5d36bedaa5f4c5a28d0fbc079cc22bc437eb562e
Author: Heitor Matsui <email address hidden>
Date: Tue May 17 15:10:35 2022 -0300

    Retry platform resize during upgrade-activate

    Due to the validation added on [1], the first attempt to resize
    the platform controllerfs may now fail as the script will resize
    backup host filesystems prior to resizing platform, thus forcing
    subsequent fs resize commands to wait for the agent to report back
    to conductor before allowing other filesystem resize commands.

    This commit makes the script retry the controllerfs-modify command
    during its validation phase if the platform size is different from
    the resized size.

    Also, because of the possible extra waiting time added until agents
    report back, the sleep time during the platform fs validation phase
    was bumped up.

    [1] https://review.opendev.org/c/starlingx/config/+/839384

    Test Plan:
    PASS: run upgrade-activate on Standard System Controller, verify
          that script runs successfully and backup/platform filesystems
          are resized accordingly
    PASS: run upgrade-activate on AIO-DX System Controller, verify that
          script runs successfully and backup/platform are resized
          accordingly

    Closes-bug: 1973817
    Change-Id: I83dc7ce365d8cd490fdfd3c6b68661082e9dd102
    Signed-off-by: Heitor Matsui <email address hidden>