Source image rejected: None of the signatures were accepted, reasons: open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory

[chandan kumar]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0 is RED in check and gate.

with latest logs https://ad4a918c918d1cf92f3d-b971135d7916cb5f7bbc35fdb125d434.ssl.cf5.rackcdn.com/855552/1/check/tripleo-ci-centos-9-content-provider/b701413/logs/undercloud/var/log/extra/rpm-list.txt

```
podman-4.2.0-3.el9.x86_64
containers-common-1-44.el9.x86_64
```
and
https://ad4a918c918d1cf92f3d-b971135d7916cb5f7bbc35fdb125d434.ssl.cf5.rackcdn.com/855552/1/check/tripleo-ci-centos-9-content-provider/b701413/logs/undercloud/home/zuul/container_image_build.log

```
ime="2022-09-02T02:03:32-04:00" level=debug msg="Pull Policy for pull [ifnewer]"\nerror creating build container: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory\nFailed to write to log, write /home/zuul/container-builds/f8866c54-53d9-435a-903c-36a38d3df83b/base/base-build.log: file already closed\n'
```

https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a adds beta keys to default-policy.json in containers-common-1-43.el9

But https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/tree/c9s there is no /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta file found.

It is breaking our container job. It all started from another bug: https://bugs.launchpad.net/tripleo/+bug/1988500

[Takashi Kajinami (kajinamit)]

Because this is a package bug, I've reported a bug to CentOS Stream 9.
 https://bugzilla.redhat.com/show_bug.cgi?id=2123611

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/b10da3f993b1be0709cfe047b292c091fa7f3554
Submitter: "Zuul (22348)"
Branch: master

commit b10da3f993b1be0709cfe047b292c091fa7f3554
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Fri Sep 2 11:02:28 2022 +0530

    Downgrade containers-common to 1-40

    containers-common-1-43 adds the new keypath[1] which will
    work with latest podman[2] which is not available in
    podman-4.1.1-6. It breaks the deployment.

    Downgrading containers-common to 1-40 fixes the issue
    till we get a new podman version.

    On release file changes, wallaby jobs are failing with
    ```
    Depsolve Error occurred: \n Problem: problem with installed package catatonit-3:0.1.7-7.el9.x86_64\n
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64
    ```
    during overcloud deployment. It blocks the above changes.

    We need to revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/853142
    the change in this patch itself and get this patch in.

    Links:
    [1]. https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a
    [2]. https://github.com/containers/image/commit/d218ff3d4611d35295615adf0913352a76684220

    Related-Bug: #1988500
    Related-Bug: #1988514
    Closes-Bug: #1985981

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ie0aea674228f011881f42b9515a2e0a73198abed

[chandan kumar (chkumar246)]

After merging https://review.opendev.org/c/openstack/tripleo-quickstart/+/855587 the jobs https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=0 are back to green, we can move it to done.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/856026

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/856026
Committed: https://opendev.org/openstack/tripleo-ansible/commit/5305f0b8a1eecc2e12a3c088ec56ffc5086852ef
Submitter: "Zuul (22348)"
Branch: master

commit 5305f0b8a1eecc2e12a3c088ec56ffc5086852ef
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Sep 6 12:32:43 2022 +0530

    Downgrade containers-common to 1-40

    containers-common-1-43 adds the new keypath[1].
    It included RPM-GPG-KEY-redhat-beta key file but forgot to include
    it in the spec file leading to above failure.

    Downgrading containers-common to 1-40 fixes the issuue.

    [1]. https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a

    Note: Putting it in directories.yml as this playbook is shared with tox
    as well as molecule jobs.

    Related-Bug: #1988514

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I0d804eeed00f0adb231ca0dc85fa57c6256ea429

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/856283
Committed: https://opendev.org/openstack/tripleo-ansible/commit/2381a7c3b246713744ab259ea8ac22be826344cb
Submitter: "Zuul (22348)"
Branch: master

commit 2381a7c3b246713744ab259ea8ac22be826344cb
Author: Cédric Jeanneret <email address hidden>
Date: Wed Sep 7 15:44:29 2022 +0200

    Ensure package is installed before downgrading

    The default provisioner doesn't seem to require the package, so it's not
    installed, so it can't downgrade. So it fails the whole molecule job.

    Related-Bug: #1988514
    Change-Id: I3a4d5705b2ab48ce7879958043827e6fdc3377ba

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/856758

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/856758
Committed: https://opendev.org/openstack/tripleo-ansible/commit/4c6889ece5b66912ae240ae222c97d726711d52a
Submitter: "Zuul (22348)"
Branch: master

commit 4c6889ece5b66912ae240ae222c97d726711d52a
Author: Cédric Jeanneret <email address hidden>
Date: Fri Sep 9 14:05:38 2022 +0200

    We need to downgrade in the pre.yml as well

    There's an issue with containers-common not providing all the needed gpg
    keys, leading to failures when the job downloads a container.

    the directories.yml kicks in too early, so the package isn't installed
    at that point, since it's installed as a dependency of podman, installed
    during the "Run bindep" task,

    Change-Id: I440615cc8fdeaf4a315dede080141528278aec94
    Related-Bug: #1988514

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/858905

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/858905
Committed: https://opendev.org/openstack/tripleo-ansible/commit/33e044d9e7af1153d6c8b0c2a12387da87789a6c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 33e044d9e7af1153d6c8b0c2a12387da87789a6c
Author: Cédric Jeanneret <email address hidden>
Date: Fri Sep 9 14:05:38 2022 +0200

    We need to downgrade in the pre.yml as well

    There's an issue with containers-common not providing all the needed gpg
    keys, leading to failures when the job downloads a container.

    the directories.yml kicks in too early, so the package isn't installed
    at that point, since it's installed as a dependency of podman, installed
    during the "Run bindep" task,

    Change-Id: I440615cc8fdeaf4a315dede080141528278aec94
    Related-Bug: #1988514
    (cherry picked from commit 4c6889ece5b66912ae240ae222c97d726711d52a)