CentOS Stream 9 wallaby overcloud & IPA image build blocker

[chandan kumar]

The team is trying to build overcloud and IPA image for stable/wallaby via testproject:

Test wallaby cs9 image build jobs | https://review.rdoproject.org/r/c/testproject/+/37713

It depends on following patches:
* 823367: Replace genisoimage with xorriso | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823367

* 823365: Preliminary work to support CentOS 9 Stream | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365

In order to get https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365 merged,

We need to make these jobs https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365/2/zuul.d/project.yaml green for stable/wallaby.

```
        - ironic-python-agent-check-image-dib-centos9:
            voting: false
        - ironic-python-agent-check-image-dib-centos9-extra:
            voting: false
```

While testing these jobs in a separate patch:
824136: [DNM] testing dib-centos9 experimental job | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/824136

We are hitting following errors while building the dib image:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d1/824136/1/check/ironic-python-agent-check-image-dib-centos9/6d1fbb6/job-output.txt

```
 echo '--- Clearing BLS entries ---'
2022-01-11 10:32:05.997692 | ubuntu-focal | 2022-01-11 10:32:05.997 | --- Clearing BLS entries ---
2022-01-11 10:32:05.997839 | ubuntu-focal | 2022-01-11 10:32:05.997 | + ls /boot/loader/entries/2621f92d81b44677be35af46f53eea68-5.14.0-39.el9.x86_64.conf
2022-01-11 10:32:05.999870 | ubuntu-focal | 2022-01-11 10:32:05.999 | /boot/loader/entries/2621f92d81b44677be35af46f53eea68-5.14.0-39.el9.x86_64.conf
2022-01-11 10:32:06.000370 | ubuntu-focal | 2022-01-11 10:32:06.000 | + rm /boot/loader/entries/2621f92d81b44677be35af46f53eea68-5.14.0-39.el9.x86_64.conf
2022-01-11 10:32:06.001745 | ubuntu-focal | 2022-01-11 10:32:06.001 | + grub2-switch-to-blscfg
2022-01-11 10:32:06.067403 | ubuntu-focal | 2022-01-11 10:32:06.067 | /usr/sbin/grub2-probe: error: cannot find a device for / (is /dev mounted?).
2022-01-11 10:32:06.074413 | ubuntu-focal | 2022-01-11 10:32:06.074 | Updating /boot/grub2/grub.cfg failed
```

It is failing at this part: https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/centos/pre-install.d/03-reset-bls-entries#L40

```
grub2-switch-to-blscfg
echo "--- Show kernels ---"
grubby --info=ALL
```

It might be a real issue while building the image. Adding this bug to track the issue and do further investigation.

OR
In order to unblock the CS9 wallaby image, we might want to remove the jobs from https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365 and merge it with a note ?

[chandan kumar (chkumar246)]

https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/824136 testing it with depends on https://review.opendev.org/c/openstack/diskimage-builder/+/824659/ .

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ac/824136/4/check/ironic-python-agent-check-image-dib-centos9/5acb0ae/job-output.txt

```
 /tmp/dib_build.7u24NpPz/hooks/root.d/10-centos-cloud-image:main:64 : curl -s https://cloud.centos.org/centos/9-stream/x86_64/images/
2022-01-17 13:31:10.418596 | ubuntu-focal | 2022-01-17 13:31:10.418 | ++ /tmp/dib_build.7u24NpPz/hooks/root.d/10-centos-cloud-image:main:64 : grep -o 'CentOS-.[^>]*GenericCloud-.[^>]*.qcow2'
2022-01-17 13:31:10.419166 | ubuntu-focal | 2022-01-17 13:31:10.419 | ++ /tmp/dib_build.7u24NpPz/hooks/root.d/10-centos-cloud-image:main:64 : head -1
2022-01-17 13:31:10.419903 | ubuntu-focal | 2022-01-17 13:31:10.419 | ++ /tmp/dib_build.7u24NpPz/hooks/root.d/10-centos-cloud-image:main:64 : sort -r
2022-01-17 13:31:10.754951 | ubuntu-focal | 2022-01-17 13:31:10.754 | sort: fflush failed: 'standard output': Broken pipe
2022-01-17 13:31:10.755932 | ubuntu-focal | 2022-01-17 13:31:10.754 | sort: write error
2022-01-17 13:31:10.761402 | ubuntu-focal | 2022-01-17 13:31:10.761 | + /tmp/dib_build.7u24NpPz/hooks/root.d/10-centos-cloud-image:main:64 : BASE_IMAGE_FILE=CentOS-Stream-GenericCloud-9-20220113.0.x86_64.qcow2
```

[chandan kumar (chkumar246)]

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ebe/824136/4/check/ironic-python-agent-check-image-dib-centos9-extra/ebe4559/job-output.txt

```
Error: Unable to find a match: sysbench genisoimage python3-virtualenv
```

[Marios Andreou (marios-b)]

Looks like this is now resolved

IPA buildimage job is back to green [1] at least in the testproject runs

Moving this one fix-released please move back if you disagree

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-9-buildimage-ironic-python-agent-wallaby