lubuntu-update-notifier provides no good way to resolve a forcibly terminated system update

[Aaron Rainbolt]

Warning, this is a very long bug report as the details are complex and the reasoning behind the resolution is just as complex. In it, I propose the addition of two new features to lubuntu-update-notifier in the Kinetic and Jammy stable releases. I realize this sounds highly questionable, however I thoroughly explain why this is helpful (and necessary) in this instance.

[ Impact ]

The lubuntu-update-notifier package is a software system that automatically checks for updates and helps the user to install them via a Qt-based GUI on Lubuntu. It is the main update notifier system in Lubuntu.

In current versions of lubuntu-update-notifier, if something happens that interrupts update installation (e.g., a power cut or a bug that forces the system to be powered down during update installation), lubuntu-update-notifier becomes unable to keep the system updated, as it has no way of resuming the interrupted update via a command such as "dpkg --configure -a". In the best case, this results in inconvenience for the user, while in the worst case, it could leave the user's system in an unusable state due to the mixture of configured and unconfigured updates on the system. It could also accidentally allow a user to fail to apply any further updates, leaving their system in a broken and vulnerable state.

While it may seem like interruptions to update installation would be very rare, there is currently a situation in Jammy at least (and possibly in other versions of Lubuntu) that always results in an interrupted update installation. Namely, until version 0.5, lubuntu-update-notifier was unable to handle the prompts that sometimes appear during updates that ask if you want to replace a configuration file, or keep your existing version. A recent SDDM update (which is still in -proposed thanks to Eickmeyer's patience with us) is known for causing this prompt to appear on Lubuntu 22.04 during a regular update procedure.

Because the version of lubuntu-update-notifier present in Lubuntu is incapable of handling these prompts, it simply hangs, theoretically indefinitely. Because the update installation inhibits system shutdown, it even makes it difficult to reboot or shut down the Lubuntu system when this happens. The user is forced to terminate the update either by killing the aptd process, or more likely, by pressing and holding their system's power button until it powers off. While the system will likely reboot into an apparently functional state, all update functionality via lubuntu-update-notifier will be broken after this happens, resulting in the problems described above.

This seems like it would be fixable by simply releasing an update to lubuntu-update-notifier that adds the functionality needed to handle the configuration conflict prompt. However, due to a race condition inherent in this situation, if lubuntu-update-notifier and a package that triggers a configuration conflict prompt (like SDDM) are both updating at the same time, the version of lubuntu-update-notifier that will be running at installation time will still be the old version that lacks the needed functionality, resulting in the hang and subsequent update interruption.

It might seem that this problem could be bypassed by using a creative postinst script in lubuntu-update-notifier to restart the update with the newer, working version of lubuntu-update-notifier. Sadly, despite numerous attempts with various creative techniques, the Lubuntu Development team was unable to get lubuntu-update-notifier to reliably configure itself before SDDM, rendering this workaround of no use. (And we tried, like, *everything* reasonable to make that work.)

This leaves only one good workaround that the Lubuntu Development team can see so far, and that is to add functionality to lubuntu-update-notifier that is capable of repairing an interrupted update. Because lubuntu-update-notifier doesn't need to be configured in order to be functional, it is sufficient for it to be simply unpacked on the system for a new version to be installed and working. When the configuration conflict prompt appears, the user can forcibly power off their system, power it back on, and then run the "Apply Full Upgrade" program again. The system will automatically repair the interrupted update and then install any new updates that may be available. By shipping a new version of lubuntu-update-notifier that can handle configuration conflict prompts, we can avoid this situation in the future, and by adding the update repair functionality, we can give users an intuitive way to unbreak their system after this series of events takes place.

This SRU backports both new features (the configuration conflict prompt handling and the interrupted update repairing) to the Kinetic and Jammy releases. It may also need to cover Focal as well, but given the very short amount of support time left in Lubuntu for Focal, it might be a better use of developer resources to skip it. Whether the backport will be done depends on what the Ubuntu SRU team believes is good, and how easy it is to backport these fixes to the older version of lubuntu-update-notifier.

[ Test plan ]

This bug is easiest to watch in action using Lubuntu Jammy. It is possible to replicate the problem in Kinetic, but it is more difficult as SDDM does not appear to generate a configuration conflict prompt in Kinetic.

For the update unbreaking feature:

Jammy/Kinetic instructions:
1. Download a Lubuntu 22.04.1 or 22.10 ISO, and install it into a virtual machine.
2. Install all available updates.
3. Reboot the machine.
4. Click the Application Menu, search for "Software", and click on "Software Sources".
5. Enter the VM's password and press Enter.
6. Click the Updates tab, and check "Pre-released updates". Then click the Close button - the system should update its software database.
7. If you're testing with Kinetic, run "sudo vim /usr/share/sddm/scripts/Xsetup", then add some text into one of the comments, save, and exit. This is necessary to get the problem to reproduce in Kinetic. **This step is unnecessary in Jammy.**
8. Click the Application Menu, and search for "Apply", then click "Apply Full Upgrade".
9. Enter the VM's password again and press Enter.
10. Wait for the upgrade to stall on "Waiting for configuration file prompt" (or something worded similarly - I can't remember the exact wording at the moment).
11. Open a terminal, and forcibly reboot the system with "systemctl reboot -i". (You can do a regular reboot if testing on Kinetic.)
12. When the VM reboots, click the Application Menu, search for "Apply", click "Apply Full Upgrade", and enter the VM's password and press Enter again.
13. You should watch the interrupted upgrade finish, and then the system search for (and if necessary, install) any new updates.

Focal instructions:
1. Download a Lubuntu 20.04.5 ISO, and install it into a virtual machine.
2. Enable -proposed.
3. Run "sudo nano /etc/apt/apt.conf.d/20apt-esm-hook.conf" and add a comment to the end of the file. This will trigger the bug.
4. Click the Application Menu, search for "Apply" and click "Apply Full Upgrade". Then enter the VM's password and press Enter.
5. Wait for the upgrade to freeze.
6. Run "sync" in a terminal, then forcibly power off the VM.
7. Boot the VM. You will *probably* run into a kernel panic, due to a new kernel being present without the initramfs for it having been generated.
8. Force poweroff the VM again, then turn it on, select "Advanced Options for Ubuntu", and boot into the older kernel normally (not using recovery mode).
9. Repeat step 4 (running "Apply Full Upgrade"). The interrupted upgrade should be repaired and, if needed, any new updates installed.

If all goes as planned, the system should be fully updated and entirely functional after this.

For the configuration conflict prompt handling feature:

Focal/Jammy/Kinetic instructions
1. Download a Lubuntu 22.04.1 or 22.10 ISO, and install it into a VM.
2. Install all available updates and reboot the VM. (NOTE: Skip this step if testing with Focal.)
3. Click on the Application Menu, search for "Software", click "Software Sources", enter your VM's password, and press Enter.
4. Click the Updates tab, check "Pre-released updates", and then click the Close button - the system should update its software database.
5. Open a terminal and run "sudo apt install lubuntu-update-notifier" to pull in the new updater from -proposed.
6. If you're testing with Kinetic, run "sudo vim /usr/share/sddm/scripts/Xsetup", then add some text into one of the comments, save, and exit. If you're testing with Focal, edit "/etc/apt/apt.conf.d/20apt-esm-hook.conf", adding an entirely new comment. This is necessary to get the problem to reproduce in Focal and Kinetic. **This step is unnecessary in Jammy.**
7. Click the Application Menu, search for "Apply", click "Apply Full Upgrade", type the VM's password, and press Enter.
8. Wait for a prompt to appear asking whether you want to keep or overwrite a configuration file.
9. Choose your desired option, and allow the upgrade to finish.
10. Check the /usr/share/sddm/scripts/Xsetup file after the update finishes. You should see a very small amount of code in this file if you chose to keep the existing file, or a rather large amount of it if you chose to overwrite the file. (On Focal, check /etc/apt/apt.conf.d/20apt-esm-hook.conf and ensure that the comment you added disappeared if you chose to overwrite the file, and that it's present if you didn't overwrite it.)

If all goes as planned, the system should be fully updated and entirely functional after this. Additionally, the file you check in step 10 should have contents that match expectations.

[ Where problems could occur ]

As this SRU introduces two new features, it is possible that a coding error in either new feature could result in any of a number of problems, such as:

- Failure to overwrite configuration files when asked to
- Overwriting of configuration files when asked not to
- Failure to repair the system after an interrupted upgrade
- Failed, hung, or otherwise messed-up updates during a normal update procedure

All of these problems are either glaringly obvious or will be easily caught using the above test plan, so it should be easy to catch these problems and fix them should they be discovered during SRU verification.

[ Other Info ]

I would like to make it very clear, that while SDDM is used extensively as part of the testing above, **SDDM has in no way caused a regression in lubuntu-update-notifier.** It simply has showed up a bug that has existed for a while and that hasn't been noticed until now since apparently updates that cause a configuration file prompt are rare on Lubuntu. **The SDDM update is *not* at fault here.** This bug could have just as easily been shown up by any other update that triggered the configuration conflict prompt (such as an update to GRUB that modified configuration files).

[Leó Kolbeinsson (leok)]

Completed both tests as per above on Lubuntu Jammy 22.04.1 with no errors..for the second test overwrite was chosen (Xsetup file).

[Launchpad Janitor (janitor)]

This bug was fixed in the package lubuntu-update-notifier - 0.5.1

---------------
lubuntu-update-notifier (0.5.1) lunar; urgency=medium

  * Added functionality for repairing an interrupted upgrade. (LP: #2002255)
  * Fixed an assumption in the config-file-conflict frontend.
  * Bumped version number in lubuntu-upgrader.

 -- Aaron Rainbolt <email address hidden> Wed, 14 Dec 2022 12:32:48 -0600

[Chris Halse Rogers (raof)]

Hello Aaron, or anyone else affected,

Accepted lubuntu-update-notifier into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lubuntu-update-notifier/0.5.1~22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Aaron, or anyone else affected,

Accepted lubuntu-update-notifier into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lubuntu-update-notifier/0.5.1~22.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Aaron Rainbolt (arraybolt3)]

Went through both test cases on Jammy, choosing to keep the existing configuration file when testing the configuration conflict prompt handling. All operations went as expected.

[Aaron Rainbolt (arraybolt3)]

It dawns on me we probably should have instructions for Kinetic since the verification process isn't exactly the same there. I have now added instructions for Kinetic to the test cases.

[Aaron Rainbolt (arraybolt3)]

Finally got the Kinetic instructions right after several attempts, then did testing for both test cases, again choosing to keep the existing configuration file for SDDM. Everything went as expected.

[Aaron Rainbolt (arraybolt3)]

Did testing on Kinetic for the second test case, but this time overwriting the config file. Everything went as expected.

Both Jammy and Kinetic are marked verification-done, however I am leaving the last verification-needed tag for now until we've made sure that the systems in question can still update properly when faced with normal updates. This will take time since I'm waiting for normal updates to be made available. Once normal updates are verified to work, the final verification-needed tag will be changed to verification-done.

[Aaron Rainbolt (arraybolt3)]

Normal updating is working just fine. Marking this as verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lubuntu-update-notifier - 0.5.1~22.10.2

---------------
lubuntu-update-notifier (0.5.1~22.10.2) kinetic; urgency=medium

  * SRU backport of 0.5.1 to the Kinetic stable release.

lubuntu-update-notifier (0.5.1) lunar; urgency=medium

  * Added functionality for repairing an interrupted upgrade. (LP: #2002255)
  * Fixed an assumption in the config-file-conflict frontend.
  * Bumped version number in lubuntu-upgrader.

lubuntu-update-notifier (0.5) lunar; urgency=medium

  * Add a frontend for config-file-conflict.
  * Update Standards-version to 4.6.1, no changes needed.

 -- Aaron Rainbolt <email address hidden> Sun, 08 Jan 2023 17:53:27 -0600

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for lubuntu-update-notifier has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lubuntu-update-notifier - 0.5.1~22.04.2

---------------
lubuntu-update-notifier (0.5.1~22.04.2) jammy; urgency=medium

  * SRU backport of 0.5.1 to the Jammy stable release.

lubuntu-update-notifier (0.5.1) lunar; urgency=medium

  * Added functionality for repairing an interrupted upgrade. (LP: #2002255)
  * Fixed an assumption in the config-file-conflict frontend.
  * Bumped version number in lubuntu-upgrader.

lubuntu-update-notifier (0.5) lunar; urgency=medium

  * Add a frontend for config-file-conflict.
  * Update Standards-version to 4.6.1, no changes needed.

 -- Aaron Rainbolt <email address hidden> Sun, 08 Jan 2023 17:56:49 -0600

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2002255

[Chris Halse Rogers (raof)]

Hello Aaron, or anyone else affected,

Accepted lubuntu-update-notifier into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lubuntu-update-notifier/0.1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Aaron Rainbolt (arraybolt3)]

Both test cases passed, also normal updates are still working on Focal with this bugfix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lubuntu-update-notifier - 0.1.1

---------------
lubuntu-update-notifier (0.1.1) focal; urgency=medium

  [ Simon Quigley ]
  * Add a frontend for config-file-conflict. (LP: #1998937)

  [ Aaron Rainbolt ]
  * Fixed an assumption in the config-file-conflict frontend. (LP: #2002255)
  * Added interrupted upgrade repair functionality.
  * Bumped version number in lubuntu-upgrader.

 -- Aaron Rainbolt <email address hidden> Tue, 17 Jan 2023 16:43:44 -0600