Fail to configure a Centos 8 machine with cloudlinux

[Andrea Biscuola]

Hi.

We use cloud-init to boot our hosting servers in an ovirt/RHEV environment. While the integration with a plain CentOS 8 works fine, cloud-init fails with an error on a CentOS 8 machine converted to cloudlinux.

Both the machines are derived from the same base image. Here the output (some informations are redacted for privacy reasons):

2020-09-09 12:57:43,385 - handlers.py[DEBUG]: start: init-network: searching for network datasources
2020-09-09 12:57:43,385 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2020-09-09 12:57:43,385 - util.py[DEBUG]: Read 16 bytes from /proc/uptime
2020-09-09 12:57:43,394 - util.py[DEBUG]: Reading from /var/lib/cloud/data/status.json (quiet=False)
2020-09-09 12:57:43,396 - atomic_helper.py[DEBUG]: Atomically writing to file /var/lib/cloud/data/status.json (via temporary file /var/lib/cloud/data/tmp4m1fs7sn) - w: [644] 463 bytes/chars
2020-09-09 12:57:43,396 - util.py[DEBUG]: Creating symbolic link from '/run/cloud-init/status.json' => '../../var/lib/cloud/data/status.json'
2020-09-09 12:57:43,396 - util.py[DEBUG]: Attempting to remove /run/cloud-init/status.json
2020-09-09 12:57:43,397 - util.py[DEBUG]: Running command ['systemd-detect-virt', '--quiet', '--container'] with allowed return codes [0] (shell=False, capture=True)
2020-09-09 12:57:43,412 - util.py[DEBUG]: Running command ['running-in-container'] with allowed return codes [0] (shell=False, capture=True)
2020-09-09 12:57:43,416 - util.py[DEBUG]: Running command ['lxc-is-container'] with allowed return codes [0] (shell=False, capture=True)
2020-09-09 12:57:43,419 - util.py[DEBUG]: Reading from /proc/1/environ (quiet=False)
2020-09-09 12:57:43,420 - util.py[DEBUG]: Read 113 bytes from /proc/1/environ
2020-09-09 12:57:43,420 - util.py[DEBUG]: Reading from /proc/self/status (quiet=False)
2020-09-09 12:57:43,421 - util.py[DEBUG]: Read 1317 bytes from /proc/self/status
2020-09-09 12:57:43,421 - util.py[DEBUG]: Reading from /proc/cmdline (quiet=False)
2020-09-09 12:57:43,421 - util.py[DEBUG]: Read 215 bytes from /proc/cmdline
2020-09-09 12:57:43,422 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2020-09-09 12:57:43,422 - util.py[DEBUG]: Read 16 bytes from /proc/uptime
2020-09-09 12:57:43,423 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg (quiet=False)
2020-09-09 12:57:43,423 - util.py[DEBUG]: Read 1275 bytes from /etc/cloud/cloud.cfg
2020-09-09 12:57:43,423 - util.py[DEBUG]: Attempting to load yaml from string of length 1275 with allowed root types (<class 'dict'>,)
2020-09-09 12:57:43,443 - util.py[DEBUG]: Reading from /etc/cloud/cloud.cfg.d/05_logging.cfg (quiet=False)
2020-09-09 12:57:43,444 - util.py[DEBUG]: Read 2057 bytes from /etc/cloud/cloud.cfg.d/05_logging.cfg
2020-09-09 12:57:43,444 - util.py[DEBUG]: Attempting to load yaml from string of length 2057 with allowed root types (<class 'dict'>,)
2020-09-09 12:57:43,452 - util.py[DEBUG]: Reading from /run/cloud-init/cloud.cfg (quiet=False)
2020-09-09 12:57:43,452 - util.py[DEBUG]: Read 39 bytes from /run/cloud-init/cloud.cfg
2020-09-09 12:57:43,452 - util.py[DEBUG]: Attempting to load yaml from string of length 39 with allowed root types (<class 'dict'>,)
2020-09-09 12:57:43,453 - util.py[DEBUG]: Attempting to load yaml from string of length 0 with allowed root types (<class 'dict'>,)
2020-09-09 12:57:43,454 - util.py[DEBUG]: loaded blob returned None, returning default.
2020-09-09 12:57:43,455 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
2020-09-09 12:57:43,457 - util.py[DEBUG]: Redirecting <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'> to | tee -a /var/log/cloud-init-output.log
2020-09-09 12:57:43,458 - main.py[DEBUG]: Logging being reset, this logger may no longer be active shortly
Cloud-init v. 19.4 running 'init' at Wed, 09 Sep 2020 12:57:43 +0000. Up 1814.73 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | <private_ip> | 255.255.248.0 | global | 56:6f:bf:b6:01:44 |
ci-info: | eth0 | True | fe80::546f:bfff:feb6:144/64 | . | link | 56:6f:bf:b6:01:44 |
ci-info: | eth1 | True | <public_ip> | 255.255.255.0 | global | 56:6f:bf:b6:01:45 |
ci-info: | eth1 | True | fe80::546f:bfff:feb6:145/64 | . | link | 56:6f:bf:b6:01:45 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | <gateway> | 0.0.0.0 | eth1 | UG |
ci-info: | 1 | <network> | 0.0.0.0 | 255.255.248.0 | eth0 | U |
ci-info: | 2 | <gateway> | 0.0.0.0 | 255.255.255.0 | eth1 | U |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 1 | fe80::/64 | :: | eth0 | U |
ci-info: | 2 | fe80::/64 | :: | eth1 | U |
ci-info: | 4 | local | :: | eth0 | U |
ci-info: | 5 | local | :: | eth1 | U |
ci-info: | 6 | ff00::/8 | :: | eth0 | U |
ci-info: | 7 | ff00::/8 | :: | eth1 | U |
ci-info: +-------+-------------+---------+-----------+-------+
2020-09-09 12:57:43,608 - stages.py[ERROR]: Unable to render networking. Network config is likely broken: No available network renderers found. Searched through list: ['eni', 'sysconfig', 'netplan']

Cloud-init is being installed from the standard CentOS 8 repositories without no further customisations. The informations are passed to cloud-init using the Ovirt/RHEV apis in the virtual machine definition, including the various ip addresses.

Regards

[Andrea Biscuola (absc)]

Adendum:

We found where the problem is; cloudlinux change the file /etc/os-release with the content:

NAME="CloudLinux"
VERSION="8.2 (Yury Malyshev)"
ID="cloudlinux"
ID_LIKE="rhel fedora centos"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CloudLinux 8.2 (Yury Malyshev)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:cloudlinux:cloudlinux:8.2:GA:server"
HOME_URL="https://www.cloudlinux.com/"
BUG_REPORT_URL="https://www.cloudlinux.com/support"

It seems that cloud-init isn't able to recognize the distribution as CentOS/RHEL anymore and fails to use sysconfig. As a test, we changed the file content back to the stock for CentOS 8.2 and the network was configured properly.

We opened a bug to the cloudlinux support to address this, mainly to be sure they are writing the proper informations in the /etc/os-release file. It's possible that cloud-init need to add an additional check to recognize a CentOS/RHEL derivative.

[Dan Watkins (oddbloke)]

The code which performs this determination is cloudinit.util.get_linux_distro (https://github.com/canonical/cloud-init/blob/master/cloudinit/util.py#L486); it looks only at the ID field. We would need to think through the consequences of using ID_LIKE: we may find that what we consider to be expected on (e.g.) a RHEL system is not the same as the person who wrote "rhel" into ID_LIKE.

cloud-init does already three levels at which it considers distros: OS families, distros, and distro variants. It feels like ID_LIKE should map onto this somehow, but it's not clear if (ID="cloudlinux", ID_LIKE="rhel ...") means that the distro is "cloudlinux" and the OS family is "redhat", or if that means that the variant is "cloudlinux" and the OS is "rhel" (and therefore the OS family is "redhat"). We would need to look at uses of ID_LIKE in the wild to determine what makes the most sense.

Unless there are a whole host of derivative distros which have this problem that we haven't heard about, my feeling is that the best resolution here is to simply add a CloudLinux distro module (which will be identical to the CentOS or Fedora ones, I expect, e.g. https://github.com/canonical/cloud-init/blob/master/cloudinit/distros/centos.py).

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3784