DC: subcloud remote install boots with ostree commit of Partial-Apply patch

[Christopher de Oliveira Souza]

Brief Description
--------------------
When a patch has been uploaded and applied, but not installed, it is in Partial-Apply state.

If a remote subcloud is installed via redfish (miniboot) at this point, it will end up running the patched software.

Severity
--------------------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
-------------------
On system controller:

sudo sw-patch upload ./22.12_NRR_INSVC.patch
sudo sw-patch apply 22.12_NRR_INSVC
sudo sw-patch query # (will show in Partial-Apply state)

Now check the ostree log of both ostree repos on the system controller:

ostree --repo=/sysroot/ostree/repo log starlingx
ostree --repo=/var/www/pages/feed/rel-22.12/ostree_repo log starlingx

This is normal. The feed repo has the new commit, but it is not 'running' yet (not in sysroot repo).

Now install a subcloud via redfish.

The problem will be seen by running the same two ostree log commands as above, but on the subcloud. You will see that the subcloud sysroot repo has the new commit, meaning the subcloud is running the patched software.

(see Logs section below for example).

Expected Behavior
---------------------
The subcloud should not be running the patched software.

Actual Behavior
--------------------
The sysroot repo on the subcloud has the patched ostree commit. The subcloud has booted running with this commit, meaning it is running the patch.

Root cause: This behaviour is caused by the subcloud doing an ostree repo pull from the system controller 'feed' repo (it has to because that repo is the only one available over http). It therefore ends up getting the patched commit.

Reproducibility
------------------
Reproducible

System Configuration
-------------------
DC

Load info (eg: 2022-03-10_20-00-07)
-------------------
N/A

Last Pass
-------------------
N/A

Timestamp/Logs
--------------------
If you examine the ostree log output on each you will see that the subcloud is running the Partial-Apply patch commit (as evidenced by the /sysroot/ostree/repo log on subcloud).

On System Controller:

[sysadmin@controller-0 ~(keystone_admin)]$ sudo sw-patch upload ./22.12_NRR_INSVC.patch
22.12_NRR_INSVC is already imported. Updated metadata only[sysadmin@controller-0 ~(keystone_admin)]$ sudo sw-patch apply 22.12_NRR_INSVC
22.12_NRR_INSVC is now in the repo[sysadmin@controller-0

~(keystone_admin)]$ ostree --repo=/var/www/pages/feed/rel-22.12/ostree_repo log starlingx
commit e92f4b31d1de37232d7bf2079d3a52556f46765c391f8680477774cb93c8028b
ContentChecksum: 372d31488fabae9d54fde4de635b94b46ead24c47cd03871765a15c1bc65af10
Date: 2022-11-23 10:05:06 +0000 Commit-id: starlingx-intel-x86-64-20221123100506commit e8294f0f26922ffbc49d9e2ae7cdb02e53c0214d68ad7b6f7e3f0e88efa00f41
ContentChecksum: f8b58b64590d0fbe552c4fd6164f70c75040af521c21f7f6fa3f40833cf59bf8
Date: 2022-11-23 09:14:09 +0000 Commit-id: starlingx-intel-x86-64-20221123091053[sysadmin@controller-0 ~(keystone_admin)]$

[sysadmin@controller-0 ~(keystone_admin)]$ ostree --repo=/sysroot/ostree/repo log starlingx
commit e8294f0f26922ffbc49d9e2ae7cdb02e53c0214d68ad7b6f7e3f0e88efa00f41
ContentChecksum: f8b58b64590d0fbe552c4fd6164f70c75040af521c21f7f6fa3f40833cf59bf8
Date: 2022-11-23 09:14:09 +0000 Commit-id: starlingx-intel-x86-64-20221123091053

On installed subcloud:

sysadmin@controller-0:~$ ostree --repo=/sysroot/ostree/repo log starlingx
commit e92f4b31d1de37232d7bf2079d3a52556f46765c391f8680477774cb93c8028b
ContentChecksum: 372d31488fabae9d54fde4de635b94b46ead24c47cd03871765a15c1bc65af10
Date: 2022-11-23 10:05:06 +0000 Commit-id: starlingx-intel-x86-64-20221123100506<< History beyond this commit not fetched >>

sysadmin@controller-0:~$ ostree --repo=/var/www/pages/feed/rel-22.12/ostree_repo log starlingx
commit e92f4b31d1de37232d7bf2079d3a52556f46765c391f8680477774cb93c8028b
ContentChecksum: 372d31488fabae9d54fde4de635b94b46ead24c47cd03871765a15c1bc65af10
Date: 2022-11-23 10:05:06 +0000 Commit-id: starlingx-intel-x86-64-20221123100506commit e8294f0f26922ffbc49d9e2ae7cdb02e53c0214d68ad7b6f7e3f0e88efa00f41
ContentChecksum: f8b58b64590d0fbe552c4fd6164f70c75040af521c21f7f6fa3f40833cf59bf8
Date: 2022-11-23 09:14:09 +0000 Commit-id: starlingx-intel-x86-64-20221123091053

Alarms
------------------
N/A

Test Activity
------------------
Developer Testing

Workaround
------------------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud/+/873369

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/873369
Committed: https://opendev.org/starlingx/distcloud/commit/2e5e2f23c547b01ef866f1ba3ae10a2820d71b9b
Submitter: "Zuul (22348)"
Branch: master

commit 2e5e2f23c547b01ef866f1ba3ae10a2820d71b9b
Author: Christopher Souza <email address hidden>
Date: Fri Feb 10 10:15:25 2023 -0300

    Check if there is an partial-applied patch when subcloud add is run

    In order to avoid the subcloud, which was added
    after a patch has been applied but not installed,
    to run the patched software, a validation
    was added and now it's not possible to add
    a subcloud when there is an partial-applied
    patch.

    Test Case:

    PASS: Install and bootstrap DC.
    PASS: Upload and apply a patch.
    PASS: Run subcloud add command and check if the
    output is "Patching operation is in progress."
    and check if no subcloud was added.

    Closes-bug: 2006939

    Signed-off-by: Christopher Souza <email address hidden>
    Change-Id: I4113523b6d27aa1f4cda89a56478a5f0833d6095