needrestart on Ubuntu 24.04 is restarting google-guest-agent startup and shutdown scrips on reinstall but they should not be restarted

[Philip Roche]

[Impact]
The recent change in Ubuntu 24.04 and needrestart which results in services being restarted automatically , details @ https://discourse.ubuntu.com/t/noble-numbat-release-notes/39890#services-restart-on-unattended-upgrade-26, has resulted in unexpected behaviour on Google GCE ubuntu 24.04 images where if the google-guest-agent package is reinstalled then needrestart will restart the `google-startup-scripts.service` and `google-shutdown-scripts.service` but these should only be run during the lifetime of the instance and should not be restarted.

[Test plan]

Boot a Noble image on GCE

rm /etc/needrestart/conf.d/google-guest-agent.conf # Remove the workaround
apt reinstall libc6
# Check that the scripts weren't restarted

[ Where problems could occur ]

Assuming no syntax error, this is fairly safe as a version of it has been shipped in the GCE images as a workaround, and the service names are pretty specific.

[Original report]
The recent change in Ubuntu 24.04 and needrestart which results in services being restarted automatically , details @ https://discourse.ubuntu.com/t/noble-numbat-release-notes/39890#services-restart-on-unattended-upgrade-26, has resulted in unexpected behaviour on Google GCE ubuntu 24.04 images where if the google-guest-agent package is reinstalled then needrestart will restart the `google-startup-scripts.service` and `google-shutdown-scripts.service` but these should only be run during the lifetime of the instance and should not be restarted.

It can be worked around by setting `NEEDRESTART_SUSPEND=1` prior to any re-install as per the needrestart man pages @ https://manpages.ubuntu.com/manpages/noble/man1/needrestart.1.html?&_ga=2.32620542.1557010342.1713949275-1490161839.1713949275#environment or by appending to the needrestart configuration `echo "\$nrconf{override_rc}{qr(^google-(shutdown|startup)-scripts\.service$)} = 0;" >> /etc/needrestart/conf.d/google-guest-agent.conf` which will disable this behaviour for any future google-guest-agent upgrade or reinstall.

This bug is to track the work of getting this added to the packaged `/etc/needrestart/needrestart.conf` configuration.

Similar to the work done for cloud-init @ https://bugs.launchpad.net/ubuntu/+source/needrestart/+bug/2059337

[Launchpad Janitor (janitor)]

This bug was fixed in the package needrestart - 3.6-8ubuntu3

---------------
needrestart (3.6-8ubuntu3) oracular; urgency=medium

  * d/p/ubuntu-mode.patch: Don't touch /run/reboot-required on kernel updates
    (LP: #2065863)
  * Ubuntu mode: disable it if restart mode has been explicitly set
    (LP: #2068543)
  * Add some inline documentation for the Ubuntu mode (LP: #2068573)
  * Don't restart the google-guest-agent service (LP: #2063442)
  * Don't restart the GH runner provisioner (LP: #2067800)
  * tests: guard against looping when failing.

 -- Simon Chopin <email address hidden> Fri, 14 Jun 2024 11:57:11 +0200

[Łukasz Zemczak (sil2100)]

Hello Philip, or anyone else affected,

Accepted needrestart into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/needrestart/3.6-7ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Catherine Redfield (catred)]

Verification for noble:

# create instance
catred@canonical:~$ gcloud compute instances create lp2063442 --image-project ubuntu-os-cloud --image ubuntu-minimal-2404-noble-amd64-v20240709 --metadata=startup-script='#! /bin/bash
date | cat >> /tmp/lp2063442
'

# remove fix
catred@lp2063442:~$ sudo rm /etc/needrestart/conf.d/google-guest-agent.conf
## add -proposed
catred@lp2063442:~$ sudo apt install needrestart
[...]
catred@lp2063442:~$ sudo reboot now

# verify needrestart is updated
catred@lp2063442:~$ apt-cache policy needrestart
needrestart:
  Installed: 3.6-7ubuntu4
  Candidate: 3.6-7ubuntu4
  Version table:
     3.6-7ubuntu4.1 100
        100 http://archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages
 *** 3.6-7ubuntu4 500
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu noble/main amd64 Packages
        100 /var/lib/dpkg/status

# validate that startup scripts did not restart
catred@lp2063442:~$ sudo journalctl -u google-startup-scripts.service
[...]
-- Boot 288657b0e0c94b4c9a0ebf316f626303 --
Jul 15 13:09:47 lp2063442 systemd[1]: Starting google-startup-scripts.service - Google Compute Engine Startup Scripts...
Jul 15 13:09:47 lp2063442 google_metadata_script_runner[738]: Starting startup scripts (version 20240213.00-0ubuntu3.1).
Jul 15 13:09:47 lp2063442 google_metadata_script_runner[738]: Found startup-script in metadata.
Jul 15 13:09:47 lp2063442 google_metadata_script_runner[738]: startup-script exit status 0
Jul 15 13:09:47 lp2063442 google_metadata_script_runner[738]: Finished running startup scripts.
Jul 15 13:09:47 lp2063442 systemd[1]: google-startup-scripts.service: Deactivated successfully.
Jul 15 13:09:47 lp2063442 systemd[1]: Finished google-startup-scripts.service - Google Compute Engine Startup Scripts.
catred@lp2063442:~$ cat /tmp/lp2063442
Mon Jul 15 13:09:47 UTC 2024

[Catherine Redfield (catred)]

Noble verification update: after posting the above I went back and tried to replicate the bug using the test plan for my peace of mind and was not able to. After some tweaking and input for others, I was able to reproduce the bug (and verify the -proposed package) using custom images:

Reproducer:
# create instance based on custom image without fix
$ gcloud compute instances create lp2063442 --image-project ubuntu-catred --image testing-ubuntu-2404-noble-amd64-v20240712 --metadata=startup-script='#! /bin/bash
date | cat >> /tmp/lp2063442
sudo apt reinstall google-guest-agent -y
date | cat >> /tmp/lp2063442
'

# not proposed version of needrestart
catred@lp2063442:~$ sudo apt-cache policy needrestart
needrestart:
  Installed: 3.6-7ubuntu4
  Candidate: 3.6-7ubuntu4
  Version table:
 *** 3.6-7ubuntu4 500
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu noble/main amd64 Packages
        100 /var/lib/dpkg/status

# startup scripts restarted
catred@lp2063442:~$ cat /tmp/lp2063442
Mon Jul 15 16:09:41 UTC 2024
Mon Jul 15 16:09:53 UTC 2024
Mon Jul 15 16:11:53 UTC 2024
Mon Jul 15 16:11:53 UTC 2024

Verification of proposed packages:
# create instance based on custom image without fix and with needrestart from -proposed
catred@canonical:~/code/cpc_packaging.extra/noble$ gcloud compute instances create lp2063442-verify --image-project ubuntu-catred --image testing-ubuntu-2404-noble-amd64-v20240715 --metadata=startup-script='#! /bin/bash
date | cat >> /tmp/lp2063442
sudo apt reinstall google-guest-agent -y
date | cat >> /tmp/lp2063442
'

# verify needrestart is in proposed
catred@lp2063442-verify:~$ sudo apt-cache policy needrestart
needrestart:
  Installed: 3.6-7ubuntu4.1
  Candidate: 3.6-7ubuntu4.1
  Version table:
 *** 3.6-7ubuntu4.1 100
        100 /var/lib/dpkg/status
     3.6-7ubuntu4 500
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu noble/main amd64 Packages

# verify startupscripts did not restart
catred@lp2063442-verify:~$ cat /tmp/lp2063442
Mon Jul 15 16:20:57 UTC 2024
Mon Jul 15 16:21:07 UTC 2024