containers-multinode victoria, ussuri, train failing in tempest w/ libvirt.libvirtError: internal error: unknown feature amd-sev-es

[wes hayutin]

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_89c/792904/2/check/tripleo-ci-centos-8-containers-multinode-victoria/89c2e52/logs/undercloud/var/log/tempest/stestr_results.html

for example

The branched and non-branched jobs will also need to be marked non-voting.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793089

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793091

[wes hayutin (weshayutin)]

all tempest jobs in multinode fail w/

tempest.exceptions.BuildErrorException: Server cb3e4660-f15a-4a90-a333-5c9f8d00d753 failed to build and is in ERROR status
    Details: {'code': 500, 'created': '2021-05-25T20:52:29Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cb3e4660-f15a-4a90-a333-5c9f8d00d753.'}

[wes hayutin (weshayutin)]

Perhaps related to:

2021-05-25 20:51:07.266 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova.compute.manager [req-2ae8bea7-2401-4b73-8f4a-5f18077b54f0 842c7e5864484d0088451538f04f60e2 5840d90f910843cf96a6a2dd86437c3a - default default] [instance: 6e2c8440-9c89-4636-a957-2aa90ecb7360] Failed to build and run instance: libvirt.libvirtError: internal error: unknown feature amd-sev-es

http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_89c/792904/2/check/tripleo-ci-centos-8-containers-multinode-victoria/89c2e52/logs/subnode-1/var/log/extra/errors.txt.txt

[wes hayutin (weshayutin)]

ah.. perhaps maybe our first centos-stream related issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1961558

[wes hayutin (weshayutin)]

2021-05-25T19:21:30+0000 DEBUG ---> Package edk2-ovmf.noarch 20200602gitca407c7246bf-5.el8 will be installed

The rpm called out in the bugzilla is installed on the failing nodes.

http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_89c/792904/2/check/tripleo-ci-centos-8-containers-multinode-victoria/89c2e52/logs/subnode-1/var/log/extra/podman/containers/nova_compute/log/dnf.log

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098

[OpenStack Infra (hudson-openstack)]

Change abandoned by "yatin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098
Reason: to re run as it failed

[Sandeep Yadav (sandeepyadav93)]

We tried to exclude the buggy rpm (edk2-ovmf-20200602gitca407c7246bf-5*) with https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098 . We exclude it, c8-stream contains two versions in parallel, so we use the lower version -4.

Moving back to edk2-ovmf -4 version helped:-

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ae1/793098/3/check/tripleo-ci-centos-8-containers-multinode-train/ae1a7f1/logs/undercloud/var/log/extra/podman/containers/nova_compute/log/dnf.log
~~~
edk2-ovmf noarch 20200602gitca407c7246bf-4.el8
~~~

[David Vallee Delisle (valleedelisle)]

After discussing with my nova colleagues, this is not happening on wallaby/master because we started parsing firmware descriptor files as part of the secure boot work [1]

The proposed workaround of excluding edk2-ovmf-20200602gitca407c7246bf-5 seems to work and we recommend that we keep it until bz1961562 [2] mentioned above has a fix released.

There's another possible workaround if this is not possible anymore and that would involved creating files under /etc/qemu on the compute nodes as mentioned in comment #16 of that same bz

  mkdir -p /etc/qemu/firmware
  touch /etc/qemu/firmware/50-edk2-ovmf-cc.json

This will create an empty file which can disable the new firmware, more details here [2], look for firmware description.

[1] https://github.com/openstack/nova/blob/b0cd985f0c09088098f74cc0cb1df616cc0ef12b/nova/virt/libvirt/host.py#L1601-L1664
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1961562
[3] https://libvirt.org/formatdomain.html#operating-system-booting

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-repos/+/793157

[OpenStack Infra (hudson-openstack)]

Change abandoned by "wes hayutin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145
Reason: mirrors

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-repos/+/793157
Committed: https://opendev.org/openstack/tripleo-repos/commit/8454634881e6155fb1d361e3754ab3972fd02f33
Submitter: "Zuul (22348)"
Branch: master

commit 8454634881e6155fb1d361e3754ab3972fd02f33
Author: Arx Cruz <email address hidden>
Date: Wed May 26 15:31:44 2021 +0200

    Add edk2 on exclude list for AppStream repo

    Due a bug on edk2 we need to exclude it from stable branches.

    Related-Bug: #1929634
    See rhbz: #1961558

    Change-Id: I87521c2f6f449f36853ea79303921b4f47bf2e96

[OpenStack Infra (hudson-openstack)]

Change abandoned by "wes hayutin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098
Reason: https://6b503f0db45c120402d7-f1ca70434eb1a6e0aabfc1f83dc7e0ab.ssl.cf5.rackcdn.com/792904/5/check/tripleo-ci-centos-8-containers-multinode-ussuri/4bc9100/

[Harald Jensås (harald-jensas)]

Still seeing this on Victoria:
https://90cd9efbaa8a81e36a90-051d59c98d54ce1c08087cd83140c470.ssl.cf2.rackcdn.com/792518/1/check/tripleo-ci-centos-8-standalone/7adfdca/logs/undercloud/var/log/containers/nova/nova-compute.log

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/1a557934659051f321bf13c58b8d9a7aefc209ef
Submitter: "Zuul (22348)"
Branch: master

commit 1a557934659051f321bf13c58b8d9a7aefc209ef
Author: Sagi Shnaidman <email address hidden>
Date: Wed May 26 08:04:52 2021 +0300

    Exclude buggy edk2-ovmf for stable branches

    Related-Bug: #1929634
    See rhbz: #1961558
    Change-Id: I5953c39fd328bbf8547d369e3fbafb2b607a02c2

[OpenStack Infra (hudson-openstack)]

Change abandoned by "wes hayutin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145
Reason: https://a642c482edd71b1415e2-f68799a09cb48eb5fff1dd8f4fcbf790.ssl.cf2.rackcdn.com/793145/2/gate/tripleo-ci-centos-8-containers-multinode/60c21d2/job-output.txt

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/26816c1b27b5411a740be6cf28555768877f98cc
Submitter: "Zuul (22348)"
Branch: master

commit 26816c1b27b5411a740be6cf28555768877f98cc
Author: Sandeep Yadav <email address hidden>
Date: Wed May 26 16:58:23 2021 +0530

    Exclude buggy edk2-ovmf for stable branches

    c8-stream currently contains two versions of edk2-ovmf, We are
    excluding edk2-ovmf-20200602gitca407c7246bf-5 so we use the
    lower version edk2-ovmf-20200602gitca407c7246bf-4.el8

    This is recommended from compute team that we keep using old version
    until bz1961562 [2] is fixed. Fix will come in libvirt[2].

    We fixed the check job using [3]. Fixing periodic jobs with this patch.

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1961562
    [2] https://github.com/libvirt/libvirt/commit/61d95a1073833ec4323c1ef28e71e913c55aa7b9
    [3] https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098

    Related-Bug: #1929634
    Change-Id: I0c9019ec31e989394836170bd4fe391a1db1f567

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793763

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/793815
Committed: https://opendev.org/openstack/tripleo-ci/commit/b0747d54d2335a5b56aed1ecbe37da42e6ea3bcf
Submitter: "Zuul (22348)"
Branch: master

commit b0747d54d2335a5b56aed1ecbe37da42e6ea3bcf
Author: Sandeep Yadav <email address hidden>
Date: Mon May 31 21:04:02 2021 +0530

    Exclude buggy edk2-ovmf for t/u/v container-build

    c8-stream currently contains two versions of edk2-ovmf, We are
    excluding edk2-ovmf-20200602gitca407c7246bf-5 so we use the
    lower version edk2-ovmf-20200602gitca407c7246bf-4.el8

    This is recommended from compute team that we keep using old version
    until bz1961562 [2] is fixed. Fix will come in libvirt[2].

    We fixed the check job using [3]. Periodic container build job don't
    utilize repos from tripleo-quickstart release file, AppStream.repo
    seems coming from configure-mirrors[4] role.

    Creating a task here to include the exclude for buggy edk2-ovmf
    version.

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1961562
    [2] https://github.com/libvirt/libvirt/commit/61d95a1073833ec4323c1ef28e71e913c55aa7b9
    [3] https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098
    [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/tasks/mirror/CentOSStream-8.yaml#L10

    Closes-Bug: #1929634
    Change-Id: I843aee754b29a49ec5a2c88af1bda7bce7a60aee

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/794737

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/794737
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/25612de3f223256e3019f72264c6d5eef2010c56
Submitter: "Zuul (22348)"
Branch: master

commit 25612de3f223256e3019f72264c6d5eef2010c56
Author: yatinkarel <email address hidden>
Date: Fri Jun 4 13:50:31 2021 +0530

    Exclude buggy edk2-ovmf for stable branches

    Followup of [1] to add exclude for buggy edk2-ovmp
    to pending c8-stream releases.

    [1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098

    Related-Bug: #1929634
    Related-Bug: rhbz#1961558
    Change-Id: I1b4b96449e95e9748a428153324e2b0bdbc5d6b1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/796235
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/8712072048a2f7218d6dab23ecbe3a980f702fed
Submitter: "Zuul (22348)"
Branch: master

commit 8712072048a2f7218d6dab23ecbe3a980f702fed
Author: Sandeep Yadav <email address hidden>
Date: Mon Jun 14 12:06:38 2021 +0530

    Remove workaround to exclude newer edk2-ovmf

    Because of bug[1] we have added workaround to pin edk2-ovmf
    by adding exclude in [1] [2] and [3]

    Fix for bug[4] is released: libvirt-6.0.0-36.el8, C8 Mirror now
    contains libvirt-7.0.0-14.1.el8.x86_64.rpm

    With this patch, removing workaround we added in [1] [2] and [3]

    [0] https://bugzilla.redhat.com/show_bug.cgi?id=1961562
    [1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145
    [2] https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098
    [3] https://review.opendev.org/c/openstack/tripleo-quickstart/+/794737
    [4] http://mirror.ca-ymq-1.vexxhost.opendev.org/centos/8/virt/x86_64/advanced-virtualization/Packages/l/

    Related-Bug: #1929634
    Change-Id: I27d040dd21ad6ccd858ce8605e49e4a59c17a64e

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Arx Cruz <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793763

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/800646

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Slagle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793089
Reason: Abandoning this patch per the TripleO Patch Abandonment guidelines
(https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html).
If you wish to have this restored and cannot do so yourself, please reach out
via #tripleo on OFTC or the OpenStack Dev mailing list.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Slagle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793091
Reason: Abandoning this patch per the TripleO Patch Abandonment guidelines
(https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html).
If you wish to have this restored and cannot do so yourself, please reach out
via #tripleo on OFTC or the OpenStack Dev mailing list.