At [1] the centos9 container build is failing because we are configuring 'runc' as the container runtime [2]. The base container is failing to build with a trace like [3]:
error running container: error from runc creating container for [/bin/sh -c dnf install -y crudini && crudini --del /etc/dnf/dnf.conf main override_install_langs && crudini --set /etc/dnf/dnf.conf main clean_requirements_on_remove True && crudini --set /etc/dnf/dnf.conf main exactarch 1 && crudini --set /etc/dnf/dnf.conf main gpgcheck 1 && crudini --set /etc/dnf/dnf.conf main install_weak_deps False && if [ 'centos' == 'centos' ];then crudini --set /etc/dnf/dnf.conf main best False; fi && crudini --set /etc/dnf/dnf.conf main installonly_limit 0 && crudini --set /etc/dnf/dnf.conf main keepcache 0 && crudini --set /etc/dnf/dnf.conf main obsoletes 1 && crudini --set /etc/dnf/dnf.conf main plugins 1 && crudini --set /etc/dnf/dnf.conf main skip_missing_names_on_install False && crudini --set /etc/dnf/dnf.conf main tsflags nodocs]: time="2021-11-02T13:59:56-04:00" level=error msg="this version of runc doesn't work on cgroups v2"
Using the hack at [4] to explicitly force 'crun' we were able to get the containers built successfully (e.g. [5])
Patches from Sagi at [6],[7] should be a better/more permanent fix
[1] https://review.rdoproject.org/zuul/build/8cc09a86e56f4fa5a09c3ca68dacb0e9
[2] https://logserver.rdoproject.org/67/36267/18/check/tripleo-build-containers-stream9-development/8cc09a8/etc/containers/containers.conf
[3] https://logserver.rdoproject.org/67/36267/18/check/tripleo-build-containers-stream9-development/8cc09a8/logs/container-builds/49051216-f3d3-4dd3-9f29-c4cc6ab074e8/base/base-build.log
[4] https://review.opendev.org/c/openstack/tripleo-ci/+/816441/3/roles/build-containers/tasks/main.yaml
[5] https://review.rdoproject.org/zuul/build/265f0c4600bb49f7a327bd82e2c132f4
[6] https://review.rdoproject.org/r/c/config/+/36545
[7] https://review.opendev.org/c/openstack/tripleo-ansible/+/816445
https:/