cloud-init EC2 userdata run and unattended upgrades dpkg lock race condition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
cloud-images |
Invalid
|
Undecided
|
Unassigned | ||
cloud-init (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
The `cloud-init` service can utilize EC2 userdata as a boot time run script
Ubuntu 16.04 introduced unattended security upgrades by default. This process is done by an apt timer systemd service which does the check at 6AM/6PM respectively
systemd utilizes parallelization (at least from what I've investigated) when doing service boot
What ends up happening is the following:
* User boots a high core CPU instance
* systemd starts a parallel boot process based on the number of cores
* During this time period there is the possibility of `cloud-init` and the apt timer service running at the same time
* The userdata script run by `cloud-init` attempts to install a package through `apt-get`
However at the same time (due to parallel service start) the unattended security updates are also trying to install updates
* The `cloud-init` script is unable to get a dpkg lock and bails out
Due to the fact that we wish not to stride away from the official images and stay with what’s supported we can’t simply bake our own images and use that (this causes an increased support burden on our side). The easiest solution I can think of in this case would be to have the `cloud-init` service depend on the apt timer completion or visa versa.
Chris, I have added cloud-init as an affected package and I'll drop cloud-images. This is a duplicate of bug #1693361 so I mark it a duplicate so this issue can be discussed in one place.