shutdown-scripts service is restarted on package upgrade - causing any configured shutdown scripts to be run

[Philip Roche]

[Impact]
========

Initially reported @ https://github.com/GoogleCloudPlatform/guest-agent/issues/119

Basically anytime the google-guest-agent package is updated, these services are restarted which means the customer scripts are run at a time they are not supposed to be. This caused an outage for one customer who had some logic in their shutdown script that basically turned off their services.

The issue is most certainly in the autogenerated preinst script which should not be there. Stopping the shutdown service will invoke the shutdown script. This wasn't there in previous Ubuntu versions.

Ubuntu 22.04 preinst script
```
#!/bin/sh
set -e
# Automatically added by dh_installdeb/13.6ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
# Automatically added by dh_installsystemd/13.6ubuntu1
if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
fi
# End automatically added section
```

Ubuntu 20.04 preinst script

```
root@ubu2004:/tmp/DEBIAN# cat preinst
#!/bin/sh
set -e
# Automatically added by dh_installdeb/12.10ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
```

One reason this issue was of concern as the latest google-guest-agent update was a rebuild promoted by golang-1.18 being updated to address CVEs prompting a rebuild of all packages dependant on golang-1.18 and publication of those rebuilds to the -security pocket. unattended-upgrades then upgraded google-guest-agent in the background which restarted the shutdown-scripts.

We do want the services to be enabled on initial installation and to be started on instance start but they should not be restarted/stopped during the lifetime of the instance.

[Test Plan]
===========

After having the fixes in, we'll run the reproducer and the test script mentioned in the first comment: https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2019089/comments/1.

This will help us validate that the fix is indeed working as it should.

A demonstration of that reproducer script is shown in the 2nd comment:
https://bugs.launchpad.net/ubuntu/+source/google-guest-agent/+bug/2019089/comments/2.

Further, we'll also build the images and run the CTF tests (internal tests) on them on our side and then ask Google to validate these images to make sure they also confirm that the bug is indeed fix and there's no regression or fallout with this update.

Once Google comes back with the validation, we'll assume enough testing and verification has been done and will mark the bug with the same.

[Where Problems Could Occur]
============================

The patch just adds "--no-stop-on-upgrade" to the already existing "--no-start --no-restart-after-upgrade". So if somebody is relying on that behavior or has a workaround that somehow, it'll break the setup for them. But this update should bring them the right fix. I don't expect a lot of users to regress because of that, though.

[Philip Roche (philroche)]

I have a patched google-guest-agent for Ubuntu 22.04 in PPA https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/

And reproducer script and logs @ https://people.canonical.com/~philroche/20230508-google-guest-agent-regression/

The reproducer confirms that the shutdown script is no longer run on upgrade.

The patch was as follows:

```
--- control/preinst
+++ control/preinst
@@ -3,8 +3,3 @@
# Automatically added by dh_installdeb/13.6ubuntu1
dpkg-maintscript-helper rm_conffile /etc/defaults/instance_configs.cfg 20200617.00-0ubuntu2\~ -- "$@"
# End automatically added section
-# Automatically added by dh_installsystemd/13.6ubuntu1
-if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
- deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
-fi
-# End automatically added section
```

This was achieved by updating the `debian/rules` files

```
--- debian/rules
+++ debian/rules
@@ -17,7 +17,7 @@

override_dh_installsystemd:
dh_installsystemd google-guest-agent.service
- dh_installsystemd --no-start --no-restart-after-upgrade google-startup-scripts.service google-shutdown-scripts.service
+ dh_installsystemd --no-start --no-restart-after-upgrade --no-restart-on-upgrade google-startup-scripts.service google-shutdown-scripts.service

override_dh_clean:
dh_clean vendor/

```

[Christian Ehrhardt  (paelzer)]

Hi Phil,
there has been a lot of confusion in those options.

You suggest:
1.
 --no-restart-after-upgrade - which actually means "no ... after" which translates to "stop it early and start it late"
2.
 --no-restart-on-upgrade - which is known to be confusing, see the man page "Note that the --no-restart-on-upgrade alias is deprecated and will be removed in compat 14. This is to avoid confusion with the --no-restart-after-upgrade option."

While you touch it you should change it in the way it hopefully can stay, which would IMHO be:

dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service

I see this only goes back to Jammy, so the options would be there already (no backport pain)
WDYT?

[Philip Roche (philroche)]

@paelzer Thanks

> I see this only goes back to Jammy

Yes this affects jammy+ only but the patch would likely go back to previous releases too as part of the SRU exception and future bug and feature releases.

I have verified your proposed change too and the resulting `preinst` remains the same

```
dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service
```

I see from man pages [bionic](https://manpages.ubuntu.com/manpages/bionic/en/man1/dh_installsystemd.1.html),[focal]( https://manpages.ubuntu.com/manpages/focal/en/man1/dh_installsystemd.1.html), [jammy](https://manpages.ubuntu.com/manpages/jammy/en/man1/dh_installsystemd.1.html), [kinetic+](https://manpages.ubuntu.com/manpages/kinetic/en/man1/dh_installsystemd.1.html) that --no-stop-on-upgrade` is available so, as you say, likely no issues with backporting.

[Philip Roche (philroche)]

As per @paelzer's suggesting the proposed patch is now

```
--- debian/rules
+++ debian/rules
@@ -17,7 +17,7 @@

 override_dh_installsystemd:
  dh_installsystemd google-guest-agent.service
- dh_installsystemd --no-start --no-restart-after-upgrade google-startup-scripts.service google-shutdown-scripts.service
-
+ dh_installsystemd --no-start --no-stop-on-upgrade google-startup-scripts.service google-shutdown-scripts.service
+
 override_dh_clean:
  dh_clean vendor/

```

[Philip Roche (philroche)]

https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/ PPA now has patched versions of google-guest-agent and I have verified that it addresses the problem for all releases.

I will now work to create merge proposals for each release and seek sponsorship of upload to -proposed.

[Philip Roche (philroche)]

WIP merge proposal for Ubuntu devel series https://code.launchpad.net/~philroche/ubuntu/+source/google-guest-agent/+git/google-guest-agent/+merge/442648

[Philip Roche (philroche)]

I have rebuilt the unpatched focal packages and published to https://launchpad.net/~philroche/+archive/ubuntu/google-guest-agent/ but this does not result in the troublesome `preinst` section like we had in jammy+

```
# Automatically added by dh_installsystemd/13.6ubuntu1
if [ -z "${DPKG_ROOT:-}" ] && [ "$1" = upgrade ] && [ -d /run/systemd/system ] ; then
 deb-systemd-invoke stop 'google-shutdown-scripts.service' 'google-startup-scripts.service' >/dev/null || true
fi
# End automatically added section
```

Could this be as a result of `dh_installdeb/12.10ubuntu1` being used on focal vs `dh_installdeb/13.6ubuntu1` on jammy and the default changing between these versions?

[Bonka (bonkazx)]

Hello @philroche, I am affected by this issue also. Is there an ETA when this fix will be deployed across please? Thanks.

[Philip Roche (philroche)]

Following much testing we now have a proposed solution all options `--no-start --no-stop-on-upgrade --no-restart-after-upgrade` which passes our reproduction tests for all releases 23.10, 23.04, 22.10, 22.04, 20.04 and 18.04.

Ubuntu 23.10 has proposed package version `20230426.00-0ubuntu2` uploaded to the -proposed pocket of the archive. This will migrate to the -release pocket shortly.

We are still working on finding a solution for 16.04.

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2

---------------
google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Tue, 16 May 2023 11:11:11 +0530

[Philip Roche (philroche)]

test-google-guest-agent-upgrade-path.sh attached to help validate upgrade path. Running this with Ubuntu 23.10 mantic image daily-ubuntu-2310-mantic-amd64-v20230508 that the install of proposed google-guest-agent `20230426.00-0ubuntu2` does not trigger the bug of the shutdown script services being restarted.

[Bonka (bonkazx)]

Hello Team, please is there any ETA when this update could be available generally on latest GCP images running on 22.04.2 LTS (Jammy Jellyfish)?
My GCP instances are still running this version: 'google-guest-agent/jammy,now 20220622.00-0ubuntu2~22.04.4 amd64 [installed]'; even after running: 'sudo apt install google-guest-agent'. Thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrew Cloke (andrew-cloke)]

@bonkazx we're working to roll this fix out as quickly as possible, but the change will need to go through the SRU (Stable Release Update) process, so it is hard to communicate an ETA at this time.

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted google-guest-agent into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~23.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted google-guest-agent into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~22.10.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted google-guest-agent into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~22.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted google-guest-agent into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~20.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted google-guest-agent into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/google-guest-agent/20230426.00-0ubuntu2~18.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andrew Cloke (andrew-cloke)]

Testing
=======
The following testing has been done on the -proposed packages:

* The restart regression has been successfully tested
* Images using the -proposed packages have been built, and the CTF tests have passed on those images
* Those images have also been shared with Google and they have successfully validated them

Adjusting verification tags accordingly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~23.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~23.04.0) lunar; urgency=medium

  * No-change rebuild for Lunar.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:07:36 +0530

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for google-guest-agent has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~22.10.0

---------------
google-guest-agent (20230426.00-0ubuntu2~22.10.0) kinetic; urgency=medium

  * No-change rebuild for Kinetic

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:11:33 +0530

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~22.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~22.04.0) jammy; urgency=medium

  * No-change rebuild for Jammy.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:13:37 +0530

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~20.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~20.04.0) focal; urgency=medium

  * No-change rebuild for Focal.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:16:13 +0530

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20230426.00-0ubuntu2~18.04.0

---------------
google-guest-agent (20230426.00-0ubuntu2~18.04.0) bionic; urgency=medium

  * No-change rebuild for Bionic.

google-guest-agent (20230426.00-0ubuntu2) mantic; urgency=medium

  * d/rules: Add --no-stop-on-upgrade for upgrade path
    to enforce no stop of the services on package upgrade.
    This has the desired side-effect of not stopping, starting or
    restarting the services as a part of the upgrade (LP: #2019089)
  * d/{rules,install}: ship gce-workload-cert-refresh.timer.

 -- Utkarsh Gupta <email address hidden> Thu, 25 May 2023 13:22:23 +0530