INSVC patch install failed

[ZhangXiao]

Brief Description
-----------------
Sometimes INSVC patch can not be installed.

Severity
--------
Major

Steps to Reproduce
------------------
sw-patch upload 22.12_GEO_REDUNDANCY.patch
sw-patch apply 22.12_GEO_REDUNDANCY
sudo sw-patch host-install controller-0

Expected Behavior
------------------
Patch is applied, installed and removed successfully

Actual Behavior
----------------
Patch is applied, installed successfully, but it fails on random nodes during removal

Reproducibility
---------------
For some ISO installation, 100%; for others, 0.

System Configuration
--------------------
WRCPENG-DC1 and WRCPENG-DC1_1

Branch/Pull Time/Commit
-----------------------
Branch and the time when code was pulled or git commit or cengn load info

Last Pass
---------
Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass.
Use this section to also indicate if this is a new test scenario.

Timestamp/Logs
--------------
2023-11-22T18:48:38: sw-patch-agent[5064]: ostree_utils.py(217): INFO: OSTree Deployment Error: return code: -6 , Output: **
OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1630:install_deployment_kernel:
 assertion failed (kernel_layout->bootcsum == bootcsum):
("f3628ac3f730917cfb034d98f4b5604decbcc1178075cf444bfee3e2d4a9ce62" ==
"4604edd0da60ff7a1d4f60ec7a419aeb68b9f576e85fc9cfe4e9e8abe7b985df")

2023-11-22T18:48:38:
 sw-patch-agent[5064]: patch_agent.py(485): ERROR: Failed to pull
changes and create deploymentduring host-install.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cgcs_patch/ostree_utils.py", line 212, in create_deployment
    subprocess.run(cmd, shell=True, check=True, capture_output=True)
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'ostree admin deploy starlingx --no-prune --retain' died with <Signals.SIGABRT: 6>.

Test Activity
-------------
[Sanity, Feature Testing, Regression Testing, Developer Testing, Evaluation, Other - Please specify]

Workaround
----------
No

[ZhangXiao (zhangxiao-windriver)]

When construction the ostree repo, ostree needs us to make a checksum(boot checksum) of kernel images and initramfs file stored in /usr/lib/ostree-boot. Each time to make a new deployment, that very checksum will be checked, once it is not correct, error occurred;

To avoid resetting target with INSVC patch installation, we just remount the /usr. While the principle is INSVC patch should not contain any modification on kernel or initramfs to avoid the checksum failure above;

The method to make the boot checksum is `cat file1 file2 file3 ... | sha256sum`, same file set with different order will lead to different checksum result. If this occurred when building INSVC patch without modification on kernel or initramfs, the boot checksum changed thus case the deploy failed.

[ZhangXiao (zhangxiao-windriver)]

Workaround:
Step1) Check the boot checksum order:
Checkout files from ostree repo's "usr/lib/ostree--boot", then check to see the order that calculate the checksum, for example:
$ostree --repo=/localdisk/deploy/ostree_repo ls starlingx /usr/lib/ostree-boot
d02755 0 751 0 /usr/lib/ostree-boot
00644 0 751 0 /usr/lib/ostree-boot/.ostree-bootcsumdirsource
-00644 0 751 179190353 /usr/lib/ostree-boot/initramfs-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a
-00644 0 751 589 /usr/lib/ostree-boot/initramfs.sig
-00644 0 751 1134 /usr/lib/ostree-boot/kernel.env
-00644 0 751 11035264 /usr/lib/ostree-boot/vmlinuz-5.10.0-6-amd64
-00644 0 751 589 /usr/lib/ostree-boot/vmlinuz-5.10.0-6-amd64.sig
-00644 0 751 9707040 /usr/lib/ostree-boot/vmlinuz-5.10.0-6-rt-amd64
-00644 0 751 589 /usr/lib/ostree-boot/vmlinuz-5.10.0-6-rt-amd64.sig
-00644 0 751 11035264 /usr/lib/ostree-boot/vmlinuz-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a
-00644 0 751 589 /usr/lib/ostree-boot/vmlinuz.sig
d00755 0 0 0 /usr/lib/ostree-boot/efi
[yow-wrcp-lx:2023-07-18_15-57-53]$files="/usr/lib/ostree-boot/initramfs-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a /usr/lib/ostree-boot/vmlinuz-5.10.0-6-amd64 /usr/lib/ostree-boot/vmlinuz-5.10.0-6-rt-amd64 /usr/lib/ostree-boot/vmlinuz-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a"
[yow-wrcp-lx:2023-07-18_15-57-53]$for f in $files; do ostree --repo=/localdisk/deploy/ostree_repo cat starlingx $f > `basename $f`; done
[yow-wrcp-lx:2023-07-18_15-57-53]$ls -l
total 206032
rw-rr- 1 xzhang1 users 179190353 Nov 29 03:48 initramfs-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a
rw-rr- 1 xzhang1 users 11035264 Nov 29 03:48 vmlinuz-5.10.0-6-amd64
rw-rr- 1 xzhang1 users 9707040 Nov 29 03:48 vmlinuz-5.10.0-6-rt-amd64
rw-rr- 1 xzhang1 users 11035264 Nov 29 03:48 vmlinuz-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a
[yow-wrcp-lx:2023-07-18_15-57-53]$cat vmlinuz-5.10.0-6-amd64 vmlinuz-5.10.0-6-rt-amd64 vmlinuz-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a initramfs-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a | sha256sum
fc48becbf32bca385d60301d0eff4be3714b9ae08233817e8ceae9726fb22f60 -
[yow-wrcp-lx:2023-07-18_15-57-53]$cat vmlinuz-5.10.0-6-rt-amd64 vmlinuz-5.10.0-6-amd64 vmlinuz-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a initramfs-b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a | sha256sum
b4aba956ef0d78b5fa1777d215df37fdf8b2418cd8405edcb5ff88e51190e21a -

Compare the sha256sum value and the suffix of the initramfs file. For the example above, we can see the order is rt kernel-> standard kernel -> vmlinuz -> initrd

Step2)
A: If the order is: standard kernel-> rt kernel -> vmlinuz -> initrd.
Just make patch as usual.

B: If the order is: rt kernel-> standard kernel -> vmlinuz -> initrd
Modify make_patch.py to fit for the ostree repo above:
$stx shell --container lat
root@stx-stx-lat-tool-ffb668489-7d2sj:/# cd $PATCH_TOOLS
root@stx-stx-lat-tool-ffb668489-7d2sj:/localdisk/designer/xzhang1/stx/cgcs-root/stx/update/sw-patch/cgcs-patch/cgcs_make_patc...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/tools/+/903090

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/tools/+/903090
Committed: https://opendev.org/starlingx/tools/commit/1275dc128f6db1109c599672aeefd4a585013a39
Submitter: "Zuul (22348)"
Branch: master

commit 1275dc128f6db1109c599672aeefd4a585013a39
Author: Zhang Xiao <email address hidden>
Date: Thu Dec 7 07:13:16 2023 -0800

    LAT: update to lat-sdk-20231206

    Update lat-sdk.sh for the meta-lat commit:
    https://github.com/Wind-River/meta-lat/commit/3ddaf92342

    Test plan:
      PASS: Run "stx-init-env --rebuild".
            Run "stx shell" to enter new builder container.
            Run "build-image -c".
      PASS: Check the boot checksum of new build ostree repos, the order
            is always standard kernel, rt kernel, vmlinuz and initrd.
      PASS: Qemu boot OK;

    Closes-Bug: 2045914

    Change-Id: I256c83d213bde76b82879711d8a0e038bcbf394d
    Signed-off-by: Zhang Xiao <email address hidden>