Subcloud install fails due to incompatible fails due to driver/firmware incompatibility

[Kyle MacLeod]

Brief Description
-----------------
There needs to be a way to set custom kernel cmdline parameter for the subcloud's miniboot ISO. In some cases the subcloud can fail to boot due to incompatible driver/firmware compatibility, where a kernel parameter can address the issue.

Severity
--------

<Critical: System/Feature is not usable due to the defect>

Steps to Reproduce
------------------
Standard orchestrated subcloud upgrade

Expected Behavior
------------------
All subclouds just upgrade without issue

Actual Behavior
----------------
Certain subclouds with specific hardware fail to install the new software and drop to a root(none):/# shell prompt

Reproducibility
---------------
100% reproducible

Last Pass
---------
No issues with 21.12 upgradescate if this is a new test scenario.

Workaround
----------
None

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/utilities/+/885769

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/885758
Committed: https://opendev.org/starlingx/distcloud/commit/5d03d8915a64655cbe2bfbee779a438d41431fd0
Submitter: "Zuul (22348)"
Branch: master

commit 5d03d8915a64655cbe2bfbee779a438d41431fd0
Author: Kyle MacLeod <email address hidden>
Date: Thu Jun 8 23:43:34 2023 -0400

    Add extra_boot_params install value for subcloud installs

    This commit adds support for the extra_boot_params install value. The
    extra_boot_params install value provides a mechanism to pass custom boot
    arguments into the main sysroot disk kernel options. The intent is to
    allow for custom, hardware-specific boot option(s) for a given subcloud.

    Note: these extra options are only applied for the post-miniboot boot,
    i.e., when booting into the main ostree-based installation. The boot
    arguments are NOT applied during the initial miniboot ISO boot.

    Multiple boot arguments can be provided by separating each argument by a
    single comma. Spaces are not allowed. Example install value (yaml snippet):

        extra_boot_params: arg1=val1,arg2=val2

    The extra_boot_params value is included in the miniboot ISO boot
    parameters in the form of 'extra_boot_params=arg1=val1,arg2=val2'.
    When the subcloud boots, the miniboot kickstart translates
    this parameter into proper grub/syslinux kernel options to be applied
    for subsequent ostree-based boots.

    Test Plan
    PASS:
    - Verify the extra_boot_params install value is translated into a boot
      parameter for initial miniboot ISO boots.
    - Verify that any spaces in the extra_boot_params value are detected and
      an error is raised, aborting the subcloud installation.
    - Verify an empty extra_boot_params value is ignored
    - Verify that functionality is not affected if the extra_boot_params
      value is not provided
    - Tested values:
        - Success:
          extra_boot_params: arg1=val1,arg2=val2
          extra_boot_params: arg1=val1
          extra_boot_params: arg1
          # extra_boot_params: arg1=val1,arg2=val2
        - Fail:
          extra_boot_params:
          extra_boot_params: arg1=val1, arg2=val2

    Partial-Bug: 2023407

    Change-Id: I183e07fbdef67cf5bac0da363b7f31d230e95bf5
    Signed-off-by: Kyle MacLeod <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/utilities/+/885769
Committed: https://opendev.org/starlingx/utilities/commit/05d5a6f9170639ea6b034eee9cfbe7c6e3bf7168
Submitter: "Zuul (22348)"
Branch: master

commit 05d5a6f9170639ea6b034eee9cfbe7c6e3bf7168
Author: Kyle MacLeod <email address hidden>
Date: Fri Jun 9 10:29:58 2023 -0400

    Add extra_boot_params boot parameters into miniboot parms

    When the --param extra_boot_params argument is given, we need to parse
    the given boot parameters and include them in the miniboot boot options
    by adding them into the BOOT_ARGS_COMMON variable

    Example:
    For '--param extra_boot_params=arg1=1,arg2=two'

    The following is added to the miniboot kernel options:
    extra_boot_params=arg1=1,arg2=two arg1=1 arg2=two

    Where 'extra_boot_params=arg1=1,arg2=two' is required for parsing by
    miniboot.cfg and 'arg1=1 arg2=two' are the actual boot parameters
    applied during boot of the miniboot bootimage.iso.

    Test Plan
    PASS:
    - Verify that extra_boot_params is parsed both the 'extra_boot_params='
      as well as separate miniboot kernel options (see above example)
    - Verify that the miniboot comes up with the kernel options in its
      /proc/cmdline (i.e., the initial miniboot ISO boots with the
      configured extra options)
    - Tested with the follow input:
        --param extra_boot_params=arg1=1,arg2=2
        --param extra_boot_params=arg1=1
        --param extra_boot_params=arg1

    Closes-Bug: 2023407
    Depends-On: https://review.opendev.org/c/starlingx/distcloud/+/885758

    Signed-off-by: Kyle MacLeod <email address hidden>
    Change-Id: Iae094d9f7746e8d2963e137ab809376d75ad1bca

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/885759
Committed: https://opendev.org/starlingx/metal/commit/5f34e2843db7d936d8e8f12f9432ad7626382f6a
Submitter: "Zuul (22348)"
Branch: master

commit 5f34e2843db7d936d8e8f12f9432ad7626382f6a
Author: Kyle MacLeod <email address hidden>
Date: Thu Jun 8 23:25:46 2023 -0400

    Translate extra_boot_params into disk boot kernel options

    The extra_boot_params install value is presented as a single boot
    parameter in the initial miniboot ISO boot. This kickstart change
    translates the install value into proper disk boot kernel options, so
    that the provided extra_boot_params are applied as boot options for the
    main /boot parameters in grub and syslinux.

    Although the extra_boot_params value must be a single string, multiple
    extra boot parameters can be specified by separating individual args
    by a comma. Example: extra_boot_params=arg1=1,arg2=2. This change splits
    the args by comma and ensures that the kernel boot options are separate
    for the main boot.

    Test Plan
    PASS:
    - Verify that extra_boot_params is parsed into separate kernel options
    - Verify that disk kernel options are applied when subcloud is installed
      (i.e., the final install boots with the configured extra options)
    - Verify comma-separated input values are translated into proper
      kernel options:
        - extra_boot_params=arg1=1,arg2=2 -> kernel options: arg1=1 arg2=2
        - extra_boot_params=arg1=1 -> kernel options: arg1=1
        - extra_boot_params=arg1 -> kernel options: arg1

    Partial-Bug: 2023407
    Depends-On: https://review.opendev.org/c/starlingx/distcloud/+/885758

    Change-Id: I8ed10f7ffe8af51ae7b77eaa398b824347a0a998
    Signed-off-by: Kyle MacLeod <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/886233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/886233
Committed: https://opendev.org/starlingx/metal/commit/9e94f1834a351bfe058e80b35f867ce435dacc5d
Submitter: "Zuul (22348)"
Branch: master

commit 9e94f1834a351bfe058e80b35f867ce435dacc5d
Author: Kyle MacLeod <email address hidden>
Date: Thu Jun 15 20:52:22 2023 -0400

    Fix ip -6 address netmask and workaround for multi-drivers-switch

    This commit fixes the missing support for bootstrap_address_prefix
    in the miniboot ip -6 address add command. We check for the provided
    prefix value parsed from the boot arguments and make sure that it is
    applied if present. Note that the bootstrap_address_prefix is a
    mandatory install value, so it will be provided. However, we leave the
    capability for it to be missing, in order to de-risk this commit.

    Additionally, a workaround is included for full support of
    multi-drivers-switch given in the boot arguments. When this argument is
    given we parse out the kernel module version and use it to replace the
    current kernel modules for ice/i40e/iavf with the modules of the given
    version.

    Test Plan
    PASS:
    - Replace miniboot.cfg at /var/miniboot/kickstart-override/miniboot.cfg
      on target lab system requiring multi-drivers-switch=cvl-2.54:
        - Using subcloud install-value
          extra_boot_params: multi-drivers-switch=cvl-2.54,
          verify that the subcloud switches to the legacy kernel modules
          and the subcloud is able to properly configure its IP address and
          perform the ostree pull operation from the system controller.
    - Install subcloud with no extra_boot_params, verify that the
      bootstrap_address_prefix is properly applied. Verify no regression.

    Closes-Bug: 2023407

    Signed-off-by: Kyle MacLeod <email address hidden>
    Change-Id: I4f3d8e2f240f2aa061de30014cf39dfb9b42a035