DC remote install with inactive load fails to update iso: No such file or directory

[srana]

Brief Description
-----------------
Failed to add subcloud with an inactive load (dcmanager subcloud add ....) - No such file or directory error during pre-install update iso.

Severity
--------
<Critical: System/Feature is not usable after the defect>

Steps to Reproduce
------------------
deploy subcloud with an inactive load

Expected Behavior
------------------
the subcloud should be installed successfully

Actual Behavior
----------------
the subcloud failed to be installed

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
DC

Branch/Pull Time/Commit
-----------------------
Master (25/08/2023)

Last Pass
---------
Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass.
Use this section to also indicate if this is a new test scenario.

Timestamp/Logs
--------------
2023-08-25 20:36:34.757 110604 INFO dccommon.subcloud_install [-] Running update_iso_cmd:
....

2023-08-25 20:36:38.877 110604 ERROR dccommon.subcloud_install [-] Failed to update iso: /usr/local/bin/gen-bootloader-iso-centos.sh
....

2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager Traceback (most recent call last):
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python3/dist-packages/dcmanager/manager/subcloud_manager.py", line 1395, in _run_subcloud_install
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager install.prep(consts.ANSIBLE_OVERRIDES_PATH, payload)
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python3/dist-packages/dccommon/subcloud_install.py", line 594, in prep
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager self.update_iso(override_path, iso_values)
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager File "/usr/lib/python3/dist-packages/dccommon/subcloud_install.py", line 424, in update_iso
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager raise Exception(msg)
2023-08-25 20:36:38.878 110604 ERROR dcmanager.manager.subcloud_manager Exception: Failed to update iso: /usr/local/bin/gen-bootloader-iso-centos.sh --input

Test Activity
-------------
Regression Testing

Workaround
----------
no workarounds

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/894224

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/894224
Committed: https://opendev.org/starlingx/metal/commit/0366f8552d5ac9784de9992b6ae98958c9a40381
Submitter: "Zuul (22348)"
Branch: master

commit 0366f8552d5ac9784de9992b6ae98958c9a40381
Author: Salman Rana <email address hidden>
Date: Thu Sep 7 16:11:29 2023 -0400

    Fix kickstarts patching

    Ostree doesn't manage the /var filesystem. Anything
    installed there during initial filesystem setup becomes
    unpatchable [1]. As a result, the kickstart install dir
    /var/www/pages/feed/rel-${platform_release}/kickstart
    is not updated according to patch changes.

    This commit changes the platform-kickstarts install paths
    to a place that ostree handles,
    /usr/share/platform-kickstarts/rel-${platform_release}
    in this case and symlinks it to
    /var/www/pages/feed/rel-${platform_release}/kickstarts.

    [1] https://review.opendev.org/c/starlingx/ha/+/890918

    Test Plan:
    1. PASS: ISO install and verify symlink created:
             /var/www/pages/feed/rel-${platform_release}/kickstarts ->
             /usr/share/platform-kickstarts/rel-${platform_release}
    2. PASS: Verify that the centos/ dir, kickstart.cfg & miniboot.cfg
             are installed to /usr/share/platform-kickstarts/
             rel-${platform_release}
    3. PASS: Verify PATCH apply, ensure that changes are applied to
             /var/www/pages/feed/rel-${platform_release}/kickstarts
    4. PASS: Manually remove/re-install the platform-kickstarts package
             and verify kickstarts dir and symlink

    Closes-Bug: 2034753
    Closes-Bug: 2035109

    Change-Id: I307d28c086bb3d9f0e4d6792db44e55c99358a50
    Signed-off-by: Salman Rana <email address hidden>

[Ghada Khalil (gkhalil)]

This code change introduced a sanity issue was reported in https://bugs.launchpad.net/starlingx/+bug/2037545

A revert has been merged:
https://review.opendev.org/c/starlingx/metal/+/896525

Re-opening this LP as the original fix will need to be re-worked.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/897328

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/897328
Committed: https://opendev.org/starlingx/metal/commit/d24e48e490acfaf361a2c7bbc28ef42faef71d20
Submitter: "Zuul (22348)"
Branch: master

commit d24e48e490acfaf361a2c7bbc28ef42faef71d20
Author: Salman Rana <email address hidden>
Date: Wed Oct 4 05:56:50 2023 -0400

    Fix kickstarts patching

    Ostree doesn't manage the /var filesystem. Anything
    installed there during initial filesystem setup becomes
    unpatchable [1]. As a result, the kickstart install dir
    /var/www/pages/feed/rel-${platform_release}/kickstart
    is not updated according to patch changes. /var/www/pages/feed/rel-${platform_release}/kickstart
    is currently only used for PXE boot installs.
    Subcloud remote installations are using the miniboot.cfg
    kickstart from the load-imported ISO
    (we may want to change this in some future commit).

    This commit adds kickstart update support to
    pxeboot-feed.service (pxeboot_feed.sh) so that
    /var/www/pages/feed/rel-${platform_release}/kickstarts
    is refreshed based on the kickstart dir from
    /ostree (i.e., the patched changes).

    [1] https://review.opendev.org/c/starlingx/ha/+/890918

    Test Plan:
    1. PASS: Verify Debian build and DC system install
             (virtual lab - disk and pxe installs)
    2. PASS: Verify pxe install (DC remote install) with
             patched kickstart
    3. PASS: Create a patch with changes to kickstart feed:
              - modify an existing kickstart
              - create a new kickstart file
              - delete an existing file
              - create a new kickstart sub-directory
              - modify centos subdir
             verify patch apply, ensure that changes are
             correctly applied to:
             /var/www/pages/feed/rel-${platform_release}/kickstarts
    4. PASS: Revert the patch from test #3 and ensure changes
             are correctly undone in the feed dir

    Closes-Bug: 2034753

    Change-Id: I74804bff23a74512db6a95fa514c84a1a6ea54a8
    Signed-off-by: Salman Rana <email address hidden>