udevadm settle failure on Azure Hirsute

[James Falcon]

When launching a daily hirsute image, more often than not, we see a failure and traceback in the cloud-init logs:

2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'filesystem': 'ext4', 'device': 'ephemeral0.1'}]
2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: ephemeral0.1 is mapped to disk=/dev/disk/cloud/azure_resource part=1
2021-09-14 16:42:40,580 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2021-09-14 16:42:40,581 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2021-09-14 16:44:40,593 - util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 120.012 seconds
2021-09-14 16:44:40,593 - util.py[WARNING]: Failed during filesystem operation
Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
2021-09-14 16:44:40,593 - util.py[DEBUG]: Failed during filesystem operation
Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 157, in handle
    util.log_time(logfunc=LOG.debug,
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 886, in mkfs
    assert_and_settle_device(device)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 765, in assert_and_settle_device
    util.udevadm_settle()
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2608, in udevadm_settle
    return subp.subp(settle_cmd)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
    raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['udevadm', 'settle']
Exit code: 1
Reason: -
Stdout:
Stderr:
2021-09-14 16:44:40,636 - handlers.py[DEBUG]: finish: init-network/config-disk_setup: SUCCESS: config-disk_setup ran successfully

We should investigate the root cause.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3904