hotplug causing cloud-init to spike CPU usage
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-init |
Fix Released
|
Critical
|
Unassigned | ||
cloud-init (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
In 21.3, we added udev rules to enable the cloud-init hotplug functionality. If a new device is detected, we call into cloud-init to see if hotplug is supported/enabled, then proceed accordingly based on the results. There are cloud users that are creating and disposing docker containers at a very high rate. This causes many virtual ethernet adapters to be created and disposed. This is triggering cloud-init events at a high volume, consuming significant CPU. Even with the hotplug functionality being disabled, the act of checking if hotplug is enabled is causing the spikes in CPU.
The path taken is:
https:/
to
https:/
to
https:/
For more context, see IRC conversations from 10/1/2021 and 10/4/2021:
https:/
https:/
Changed in cloud-init: | |
status: | Triaged → Fix Committed |
As far as a fix goes, I'm leaning towards not including the udev rule during install, then installing it during our normal boot process if we detect that hotplug has been enabled.
Another possible solution is to modify the script called by the udev event to only trigger if we detect a PCI device, but IIRC that won't work on all clouds as some clouds expose their new devices as virtual devices.