unattended-upgrade of nova-common failure due to conffile prompt

[Trent Lloyd]

unattended-upgrades attempted to upgrade nova from 2:17.0.9-0ubuntu1 to 2:17.0.10-0ubuntu2.1 (bionic-security), however nova-common contains a modified conffile (/etc/nova/nova.conf) which prompts during upgrade and leaves apt/dpkg in a permanent error state requiring manual intervention. It also prevents other automated apt install operations from working while in this state.

I understand that this conffile prompt is a generally known problem and that unattended-upgrades specifically attempts to skip upgrades that have such a conffile prompt, however that did not work on this case. I am filing this bug to try and identify and resolve the cause and this affected multiple systems in an Ubuntu OpenStack deployment.

rbalint advised that this is very likely a more complex interaction with the exact upgrades that were being staged at the time and hence more logs would be needed, indeed attempting to reproduce this very simply with a downgrade of nova packages to 2:17.0.0-0ubuntu1 results in it being skipped, as expected:
root@juju-c21ec6-bionic-nova-7:/home/ubuntu# unattended-upgrade
Package nova-common has conffile prompt and needs to be upgraded manually

And from the unattended-upgrades log we can see that 179 packages in total were scheduled to upgrade together during this run.

Attaching the following logs files:
/var/log/unattended-upgrades/*
/var/log/dpkg*
dpkg_-l (As at 2020-04-27 16:22, the same time period as the unattended-upgrades logs, but the dpkg.log* files were taken later but also cover the full time period from before 2019-12-28 and after 2020-04-27).

The first instance of the failure is in unattended-upgrades.log.4.gz Line 161
"2019-12-28 06:15:29,837 Packages that will be upgraded: amd64-microcode... [truncated, 179 packages total]"

That relates to the output in unattended-upgrades-dpkg.log.4.gz Line 791 "Log started: 2019-12-28 06:25:56"

Which relates to the output of dpkg.log.6.gz Line 392
"2019-12-28 06:25:56 upgrade nova-compute-kvm:all 2:17.0.9-0ubuntu1 2:17.0.10-0ubuntu2.1"

It fails many times after that as anytime you attempt to install a package, it attempts to configure nova.conf again and exits with an error again. But that is the original failure. But note that various package upgrades happened by unattended-upgrades (and possibly other sources) in the intervening 4 months and so I guess reproducing the situation may require reverse engineering the original package list from the dpkg logs. I have not currently attempted to do that with the hopes intimate knowledge of the unattended-upgrades code and logs will make that process faster.

A full sosreport from the system is available if more information is required that will include other log files, and various other command outputs. It is not uploaded initially for privacy.

[Trent Lloyd (lathiat)]

Uploaded all historical log files in lp1893889-logs.tar.gz
Uploaded dpkg_-l

For convenient access also uploaded unattended-upgrades.log.4, unattended-upgrades-dpkg.log.4 and dpkg.log.6 which have the lines from the first instance of hitting the error

[Balint Reczey (rbalint)]

As I recall the unattended-upgrades package which hit this bug was not the latest version from bionic-updates. Right?

[Trent Lloyd (lathiat)]

Right, the systems are running 1.1ubuntu1.18.04.11 - in my original query to you I was trying to figure out if the patches in .12 or .13 were likely to have caused this specific situation and you weren't sure hence the bug report with more details.