removing kernels should not require a restart afterward

[Daniel Barrett]

[Impact]

The rationale behind the SRU to Xenial is that with latest unattended-upgrades SRU it starts removing unused kernels, but all older kernels are not removed in a single run. With update-notifier and u-u not fixed they place /var/run/reboot-required asking for a reboot when it is not needed.

[Test Case]

1. Perform a kernel upgrade normally via "apt-get dist-upgrade".
2. Reboot.
3. Run "apt-get autoremove" to delete the old kernel packages.
4. "System Notification Helper" now reports that the computer requires a reboot.

The "autoremove" operation shouldn't require a reboot, logically speaking, because it's just removing files that are unused by the OS.

[ Regression Potential ]

If the check for skipping placing the /var/run/reboot-required file is too broad it may make kernel upgrades fail to ask for reboot. The fix changes a hook called by maintainer scripts and a failure in the hook can make kernel package installations fail.
The fix is simple and was tested in several releases thus regressing in these ways is unlikely.

[ Original Bug Text ]

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: apt 1.0.1ubuntu2.7
ProcVersionSignature: Ubuntu 3.13.0-53.89-generic 3.13.11-ckt19
Uname: Linux 3.13.0-53-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
CurrentDesktop: KDE
Date: Sat May 23 12:47:15 2015
InstallationDate: Installed on 2013-08-31 (629 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
SourcePackage: apt
UpgradeStatus: Upgraded to trusty on 2014-04-26 (391 days ago)
---
ApportVersion: 2.14.1-0ubuntu3.11
Architecture: amd64
CurrentDesktop: KDE
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=UUID=66f11ff7-00bb-4452-9168-003cf9078308
InstallationDate: Installed on 2013-08-31 (632 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
MachineType: System manufacturer System Product Name
Package: linux (not installed)
ProcFB: 0 nouveaufb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-53-generic root=UUID=02741f1f-8107-4a0f-b9a6-31ef470b1389 ro libata.force=noncq quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 3.13.0-53.89-generic 3.13.11-ckt19
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-53-generic N/A
 linux-backports-modules-3.13.0-53-generic N/A
 linux-firmware 1.127.12
RfKill:

Tags: trusty
Uname: Linux 3.13.0-53-generic x86_64
UpgradeStatus: Upgraded to trusty on 2014-04-26 (395 days ago)
UserGroups: adm cdrom dialout dip fuse lightdm lpadmin plugdev sambashare sudo
WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 08/12/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 4210
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P9X79
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr4210:bd08/12/2013:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnP9X79:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

[Brian Murray (brian-murray)]

Entries for kernel versions appear in the grub boot menu and as such a reboot is required for the those kernel versions to disappear. I guess strictly speaking this isn't necessary and it looks like the control scripts for the Ubuntu kernel would need modifying to make this change.

The postinst file touches /usr/share/update-notifier/notify-reboot-required and maybe one of the rm scripts calls it.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1458204

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Daniel Barrett (dbarrett-m)]

apport information

[Cavsfan (cavsfan)]

I could not successfully run the command.

[Jarno Suni (jarnos)]

When removing a linux-image-extra package, kernel post-installation scripts including /etc/kernel/postinst.d/update-notifier will be run. That is a symbolic link to /usr/share/update-notifier/notify-reboot-required, which can be modified to fix this bug.

[Steve Langasek (vorlon)]

Why does *removing* a linux-image-extra package cause post-*installation* scripts to be run? This sounds like a bug on the kernel side, to me.

[Jarno Suni (jarnos)]

Yes, it is odd. I think the post-installation scripts will be run because the depending linux-image-<release> package might not get removed (and is present at the time anyway), thus initramfs-tools is used to create /boot/initrd.img-<release> without the extra kernel modules provided by linux-image-extra-<release>. I guess that exception could be handled in /etc/kernel/postrm.d/initramfs-tools alternatively.

Nowadays also /etc/kernel/postinst.d/unattended-upgrades touches /var/run/reboot-required so, that should be modified, as well (if post-installation script will be run).

BTW running the post-installation scripts then may cause other problem, too: Bug #1678187

[Jarno Suni (jarnos)]

Steve Langasek, such a bug has been reported (Bug #1501884)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jarno Suni (jarnos)]

An exception for linux-image-extra packages can be added in /etc/kernel/postinst.d/update-notifier and in /etc/kernel/postinst.d/unattended-upgrades

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "code to add before touching /var/run/reboot-required in post-installation scripts" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Jarno Suni (jarnos)]

An exception for linux-image-extra packages can be added in /etc/kernel/postinst.d/update-notifier and in /etc/kernel/postinst.d/unattended-upgrades

Also escape '+' characters, as package name may contain those:
https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Source

[Jarno Suni (jarnos)]

The code could be added to /etc/kernel/postinst.d/zz-update-grub, too, to avoid unnecessarily updating grub.

[Jarno Suni (jarnos)]

Because the code can be applied in many kernel post-installation scripts, I have attached it in Bug #1501884 and further updates go there.

[Julian Andres Klode (juliank)]

Fixed that for now in unattended-upgrades and update-notifier by adding

case "$DPKG_MAINTSCRIPT_PACKAGE::$DPKG_MAINTSCRIPT_NAME" in
    linux-image-extra*::postrm)
        exit 0;;
esac

We can then still decide if we want to run postinst.d scripts in linux-image-extra removals or not, but let's use bug 1501884 for that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.98ubuntu4

---------------
unattended-upgrades (0.98ubuntu4) bionic; urgency=medium

  * Cherry pick from git:
    - Fix version of test package in test_remove_unused_dependencies (Closes: #886803)

 -- Julian Andres Klode <email address hidden> Fri, 12 Jan 2018 10:10:29 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.189

---------------
update-notifier (3.189) bionic; urgency=medium

  * Do not notify-reboot-required on linux-image-extra removal (LP: #1458204)

 -- Julian Andres Klode <email address hidden> Fri, 12 Jan 2018 09:51:27 +0100

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie artful. The bug task representing the artful nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Brian Murray (brian-murray)]

Hello Daniel, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Barry Kolts (bhkolts)]

Brian,

On Dec 3rd I installed from xenial-proposed version 1.1ubuntu1.18.04.7~16.04.0.
On Dec 4th unattended-upgrades installed new kernel and rebooted, as it should.
Today, Dec 5 unattended-upgrades removed old kernels, which it should do, and then rebooted which it shouldn't do. I have attached unattended-upgrades.log for the relevant days.

On a side note unattended-upgrades emailed me after it installed the new kernel on Dec 4th, as it should, but did not email me after removing old kernels on Dec 5th. It should have.

Let me know if I can provide any other information or do any further testing. My system is Ubuntu 16.04.5 LTS server.

Barry

[Balint Reczey (rbalint)]

@bhkolts: Thank you for your feedback. Linux-image-extra-4.4.0-138-generic.postrm still runs /etc/kernel/postinst.d/update-notifier that places /var/run/reboot-required, the file u-u acts upon and reboots.

Update-notifier is fixed in Bionic, but I'm hereby nominating the fix for Xenial.

[Balint Reczey (rbalint)]

No, I can't nominate it. :-)
Brian, could you please do nominate this for u-n in Xenial?

[Balint Reczey (rbalint)]

It looks like Brian can't do it either, but I uploaded the fix for update-notifier in Xenial.

[Timo Aaltonen (tjaalton)]

Hello Daniel, or anyone else affected,

Accepted update-notifier into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/update-notifier/3.168.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Barry Kolts (bhkolts)]

@tjaalton

My 16.04 server doesn't have update-notifier installed but does have update-notifier-common installed. If I install update-notifier from propose it will install a slew of other packages. If I install update-notifier-common from propose it just installs the new update-notifier-common version 3.168.10. So should i test update-notifier-common instead of update-notifier?

[Balint Reczey (rbalint)]

@bhkolts

Please just upgrade update-notifier-common. Timo's automated notification email referred to the source package from which both binary packages are built.

[Łukasz Zemczak (sil2100)]

Hello Daniel, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jarno Suni (jarnos)]

I can verify the update-notifier 3.168.10
Did not test unattended-upgrades
Did not change the tag since it is the same for unattended-upgrades unfortunately.

[Jarno Suni (jarnos)]

Verified unattended-upgrades 1.1ubuntu1.18.04.7~16.04.1

[Barry Kolts (bhkolts)]

I also can verify 1.1ubuntu1.18.04.7~16.04.1. Yesterday new kernels were installed and a reboot performed. Today old kernel was removed and no reboot performed. This test also used the new update-notifier-common version 3.168.10.

Yesterday I received an email telling me that new kernels had been installed and a reboot was performed, as it should, but today I received no email stating the old kernel had been removed. I expected one. Is this a new bug?

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-notifier - 3.168.10

---------------
update-notifier (3.168.10) xenial; urgency=medium

  [ Julian Andres Klode ]
  * Do not notify-reboot-required on linux-image-extra removal (LP: #1458204)

 -- Balint Reczey <email address hidden> Fri, 07 Dec 2018 11:14:24 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for update-notifier has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Daniel, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.7~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Balint Reczey (rbalint)]

@bhkolts: Yes, not sending an email when only kernel autoremovals took place is a regression - or an incomplete part of the kernel autoremoval. It is fixed in 19.04, but the fix is somewhat intrusive thus it may not be back-ported.

[Balint Reczey (rbalint)]

Tested 1.1ubuntu1.18.04.7~16.04.2 on Xenial.

root@x-uu-lp-1260041:~# yes no |apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-4.8.0-53-generic linux-image-extra-4.8.0-53-generic
0 upgraded, 0 newly installed, 2 to remove and 10 not upgraded.
After this operation, 234 MB disk space will be freed.
Do you want to continue? [Y/n] Abort.
root@x-uu-lp-1260041:~# ls /var/run/reboot*
ls: cannot access '/var/run/reboot*': No such file or directory
root@x-uu-lp-1260041:~# apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  linux-image-4.8.0-53-generic linux-image-extra-4.8.0-53-generic
0 upgraded, 0 newly installed, 2 to remove and 10 not upgraded.
After this operation, 234 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 53554 files and directories currently installed.)
Removing linux-image-extra-4.8.0-53-generic (4.8.0-53.56~16.04.1) ...
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
update-initramfs: Generating /boot/initrd.img-4.8.0-53-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
Removing linux-image-4.8.0-53-generic (4.8.0-53.56~16.04.1) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
update-initramfs: Deleting /boot/initrd.img-4.8.0-53-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.8.0-53-generic /boot/vmlinuz-4.8.0-53-generic
root@x-uu-lp-1260041:~# ls /var/run/reboot*
ls: cannot access '/var/run/reboot*': No such file or directory
root@x-uu-lp-1260041:~# dpkg -l unattended-upgrades | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-==========================-============-===========================================
ii unattended-upgrades 1.1ubuntu1.18.04.7~16.04.2 all automatic installation of security upgrades

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 1.1ubuntu1.18.04.7~16.04.2

---------------
unattended-upgrades (1.1ubuntu1.18.04.7~16.04.2) xenial; urgency=medium

  * Don't check blacklist too early and report updates from not allowed origins
    as kept back. (LP: #1781176)
  * test/test_blacklisted_wrong_origin.py: Fix and enable test
  * Filter out progress indicator from dpkg log (LP: #1599646)
  * Clear cache when autoremoval fails (LP: #1779157)
  * Find autoremovable kernel packages using the patterns in APT's way
    (LP: #1815494)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.1) xenial; urgency=medium

  * Start service after systemd-logind.service to be able to take inhibition
    lock (LP: #1806487)
  * Handle gracefully when logind is down (LP: #1806487)

unattended-upgrades (1.1ubuntu1.18.04.7~16.04.0) xenial; urgency=medium

  * Backport to Xenial (LP: #1702793)
  * Revert to build-depending on debhelper (>= 9~) and dh-systemd
  * Revert configuration example changes to avoid triggering a debconf question
  * debian/postinst: Update recovery to be triggered on Xenial's package versions

unattended-upgrades (1.1ubuntu1.18.04.7) bionic; urgency=medium

  * Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
    Performing upgrades in service's ExecStop did not work when the upgrades
    involved restarting services because systemd blocked other stop/start
    actions making maintainer scripts time out and be killed leaving a broken
    system behind.
    Running unattended-upgrades.service before shutdown.target as a oneshot
    service made it run after unmounting filesystems and scheduling services
    properly on shutdown is a complex problem and adding more services to the
    mix make it even more fragile.
    The solution of monitoring PrepareForShutdown() signal from DBus
    allows Unattended Upgrade to run _before_ the jobs related to shutdown are
    queued thus package upgrades can safely restart services without
    risking causing deadlocks or breaking part of the shutdown actions.
    Also ask running unattended-upgrades to stop when shutdown starts even in
    InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
    UnattendedUpgradesShutdown class. (LP: #1778219)
  * Increase logind's InhibitDelayMaxSec to 30s. (LP: #1778219)
    This allows more time for unattended-upgrades to shut down gracefully
    or even install a few packages in InstallOnShutdown mode, but is still a
    big step back from the 30 minutes allowed for InstallOnShutdown previously.
    Users enabling InstallOnShutdown node are advised to increase
    InhibitDelayMaxSec even further possibly to 30 minutes.
    - Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
      changes
  * Ignore "W503 line break before binary operator"
    because it will become the best practice and breaks the build
  * Stop using ActionGroups, they interfere with apt.Cache.clear()
    causing all autoremovable packages to be handled as newly autoremovable
    ones and be removed by default. Dropping ActionGroup usage does not slow
    down the most frequent case of not having anything to upgrade a...

[Emanuel (semanuelgr86)]

I fixed doing this (UBUNTU 22.04 - Kernel 6.2.0-1019-azure

   apt purge update-notifier-common
   apt autoremove
   rm /etc/update-notifier -r
   apt install update-notifier-common
   apt install ubuntu-server