Automatic configuration of TSX flag on cmdline

[David Vallee Delisle]

Description
-----------

Fast-forward upgrade from OSP-13 (RHEL-7.9) to OSP-16.2 (RHEL-8.3)
fails[1] during live migration with:

    [...] libvirt.libvirtError: operation failed: guest CPU doesn't
    match specification: missing features: hle,rtm

The failure is due to RHEL-8.3 (destination host) disabling an Intel
"TSX". And disabling TSX disables the 'hle' and 'rtm' features.

This was discovered during OSP fast-forward upgrades testing[+] where a
guest was being live-migrated from RHEL-7.9 (with TSX=on) to RHEL-8.3
(breaking change: TSX=off), and the migration failed with the
above-mentioned error.

[+] https://bugzilla.redhat.com/show_bug.cgi?id=1921070#c14 — Live
    migration during OSP16.2 hybrid state from RHEL7.9 to RHEL8.3 not
    working

Why?
----

RHEL-8.3 kernel disabled Intel TSX by default, because it is considered
a potential security risk:

    https://bugzilla.redhat.com/show_bug.cgi?id=1828642
    kernel: Disable Intel TSX by default on newer CPUs

Still, it is not acceptable for RHEL-8.3 kernel to break user-space in a
minor RHEL release. (See also:
https://bugzilla.redhat.com/show_bug.cgi?id=1921070#c16)

Workaround for OSP upgrades
---------------------------

This is unpalatable, but unfortunately there's no other option currently:

(1) have a TripleO config attribute that will enable TSX on the
    destination RHEL-8.3 host; set the following in /etc/default/grub:

        GRUB_CMDLINE_LINUX_DEFAULT="[...] tsx=on"

    ... and reboot the 8.3 host;

(2) live-migrate the guests from RHEL-7.9 to the RHEL-8.3;

(3) now turn off TSX on the RHEL-8.3 host kernel command-line;
    shutdown the guests;

(4) reboot the 8.3 host again, and start the guests

https://bugzilla.redhat.com/show_bug.cgi?id=1923165
https://bugzilla.redhat.com/show_bug.cgi?id=1921070

[OpenStack Infra (hudson-openstack)]

Change abandoned by "David Vallee Delisle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/783969
Reason: After discussing this, we agreed to move this to a validation with a hard stop if operators didn't have explicitly added TSX kernel flag to their KernelArgs during update/upgrade

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/790806

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/791089

[OpenStack Infra (hudson-openstack)]

Change abandoned by "David Vallee Delisle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/775729
Reason: https://review.opendev.org/c/openstack/python-tripleoclient/+/791089/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/791089
Committed: https://opendev.org/openstack/python-tripleoclient/commit/050c9aa99fc6a3818ae37fe6382318be9f59706d
Submitter: "Zuul (22348)"
Branch: stable/train

commit 050c9aa99fc6a3818ae37fe6382318be9f59706d
Author: David Vallee Delisle <email address hidden>
Date: Thu May 13 03:47:47 2021 +0000

    [train-only] post stack creation tsx validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This is explained in detail in this article [a]

    If operators don't want to add the TSX flag to the KernelArgs,
    they can always set "ForceNoTsx" to true.

    Adding this mandatory validation right after the stacks are
    updated is probably the earliest place where we can validate
    and fail if necessary. We'd rather fail quickly than too late
    as this will provide the best experience for our users.

    In addition to this, there's a tripleo-validation [b] in the
    work.

    This is meant to be train-only for now but we will have to
    refactor if (when?) we support FFU from queens to Wallaby+

    [a] https://access.redhat.com/solutions/6036141
    [b] https://review.opendev.org/c/openstack/tripleo-validations/+/790806

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: I35246fbf74394f6e315973283464085d2aef08b2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/793927

[OpenStack Infra (hudson-openstack)]

Change abandoned by "David Vallee Delisle <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/793927

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 12.6.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/790806
Committed: https://opendev.org/openstack/tripleo-validations/commit/ede25c3e36a751daf68ce151521b371bb25f50dc
Submitter: "Zuul (22348)"
Branch: master

commit ede25c3e36a751daf68ce151521b371bb25f50dc
Author: David Vallee Delisle <email address hidden>
Date: Wed May 19 04:05:38 2021 +0000

    Compute TSX validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This also impacts upstream CentOS systems.

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: Icfcfb1c07bbfbe05d27d67187d941c0c34fad2b2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/795632

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/795793

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/795794

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 15.0.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/793927
Committed: https://opendev.org/openstack/tripleo-validations/commit/74c30cf49147dedad2944da332b58079bc703200
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 74c30cf49147dedad2944da332b58079bc703200
Author: David Vallee Delisle <email address hidden>
Date: Wed May 19 04:05:38 2021 +0000

    Compute TSX validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This also impacts upstream CentOS systems.

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: Icfcfb1c07bbfbe05d27d67187d941c0c34fad2b2
    (cherry picked from commit ede25c3e36a751daf68ce151521b371bb25f50dc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/795793
Committed: https://opendev.org/openstack/tripleo-validations/commit/7c937a03069ea939f87952f3532a42e0227c9c4b
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7c937a03069ea939f87952f3532a42e0227c9c4b
Author: David Vallee Delisle <email address hidden>
Date: Wed May 19 04:05:38 2021 +0000

    Compute TSX validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This also impacts upstream CentOS systems.

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: Icfcfb1c07bbfbe05d27d67187d941c0c34fad2b2
    (cherry picked from commit ede25c3e36a751daf68ce151521b371bb25f50dc)
    (cherry picked from commit 74c30cf49147dedad2944da332b58079bc703200)
    (cherry picked from commit 1d8c110b52bf90db9efaf55b4f071a1233270163)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/795632
Committed: https://opendev.org/openstack/tripleo-validations/commit/5207e0ede4ca8950d69cd3a264a9d2efa36bb94d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 5207e0ede4ca8950d69cd3a264a9d2efa36bb94d
Author: David Vallee Delisle <email address hidden>
Date: Wed May 19 04:05:38 2021 +0000

    Compute TSX validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This also impacts upstream CentOS systems.

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: Icfcfb1c07bbfbe05d27d67187d941c0c34fad2b2
    (cherry picked from commit ede25c3e36a751daf68ce151521b371bb25f50dc)
    (cherry picked from commit 74c30cf49147dedad2944da332b58079bc703200)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/795794
Committed: https://opendev.org/openstack/tripleo-validations/commit/1111c6d617289103fe4d3b2800e312066c27a714
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1111c6d617289103fe4d3b2800e312066c27a714
Author: David Vallee Delisle <email address hidden>
Date: Wed May 19 04:05:38 2021 +0000

    Compute TSX validation

    RHEL-8.3 kernel disabled the Intel TSX (Transactional
    Synchronization Extensions) feature by default as a preemptive
    security measure, but it breaks live migration from RHEL-7.9
    (or even RHEL-8.1 or RHEL-8.2) to RHEL-8.3.

    Operators are expected to explicitly define the TSX flag in
    their KernelArgs for the compute role to prevent live-migration
    issues during the upgrade process.

    This also impacts upstream CentOS systems.

    Co-Authored-By: Martin Schuppert <email address hidden>
    Related: https://bugzilla.redhat.com/1923165
    Closes-Bug: #1916758
    Change-Id: Icfcfb1c07bbfbe05d27d67187d941c0c34fad2b2
    (cherry picked from commit ede25c3e36a751daf68ce151521b371bb25f50dc)
    (cherry picked from commit 74c30cf49147dedad2944da332b58079bc703200)
    (cherry picked from commit 1d8c110b52bf90db9efaf55b4f071a1233270163)
    (cherry picked from commit 3c80b58b45625de00bc6d6fca8fe74af24f4690e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 14.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 13.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations 12.3.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations train-eol release.