CI issue: Failed to get properties: Message recipient disconnected from message bus without replying

[Martin Kalcok]

Our CI/tests are failing due to a bug in cloud-init [0]. It occurs if we try to run `cloud-init status --wait`[1] and dbus is not fully up yet. Fix was released in cloud-init 23.4.1 [2] but I think we need a workaround before this fixed version gets released into the package repository. Since it's a race condition, it is not 100% reproducible, but in my experience it occurs more often than not.

Reproducer:

$ make check-system

Example of error traceback:

basic_cluster.bats
 ✗ setup_file failed
   Launching microovn-basic-cluster-2
   Launching microovn-basic-cluster-3
   Waiting for microovn-basic-cluster-1 to be ready
   Cleaning up microovn-basic-cluster-1
   Cleaning up microovn-basic-cluster-2
   Cleaning up microovn-basic-cluster-3
   (from function `lxc_exec' in file tests/test_helper/lxd.bash, line 35,
    from function `wait_containers_ready' in file tests/test_helper/lxd.bash, line 17,
    from function `setup_file' in test file tests/test_helper/setup_teardown/basic_cluster.bash, line 10)
     `wait_containers_ready $TEST_CONTAINERS' failed
   ..........................................................Traceback (most recent call last):
     File "/usr/bin/cloud-init", line 33, in <module>
       sys.exit(load_entry_point('cloud-init==23.3.3', 'console_scripts', 'cloud-init')())
     File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 1080, in main
       retval = util.log_time(
     File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2833, in log_time
       ret = func(*args, **kwargs)
     File "/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 126, in handle_status_args
       details = get_status_details(paths)
     File "/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 310, in get_status_details
       systemd_status = _get_systemd_status()
     File "/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 212, in _get_systemd_status
       stdout = subp.subp(
     File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
       raise ProcessExecutionError(
   cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
   Command: ['systemctl', 'show', '--property=ActiveState,UnitFileState,SubState,MainPID', 'cloud-final.service']
   Exit code: 1
   Reason: -
   Stdout:
   Stderr: Failed to get properties: Message recipient disconnected from message bus without replying

[0] https://github.com/canonical/cloud-init/issues/4676
[1] https://github.com/canonical/microovn/blob/f26397d2ef732479d62a29f5a00a7144a6dc4e98/tests/test_helper/lxd.bash#L17
[2]https://github.com/canonical/cloud-init/releases/tag/23.4.1

[Martin Kalcok (martin-kalcok)]

The problem seems to have gone away for now but I'll keep this open. Hopefully it wont resurface until we'll switch to 24.04 containers for the tests. 24.04 should have version of cloud-init that is not affected by this bug.