`google-startup-scripts.service` never runs in lunar

[Chloé Smith]

[Impact]

`google-guest-agent` is provided by Google for installation within guests that run on GCE. It's just one of a collection of tools and daemons that ensure that the Ubuntu images published to GCE run properly on their platform.

A recent bugfix (LP: #1998864) has caused a regression where `google-startup-scripts.service` is never deployed. The effect of this is that lunar lobster images never deploy any start-up scripts that can be provided in the instance's metadata before boot.

Unfortunately, better knowledge of `systemd` and was needed to foresee that simply removing the (mistakenly) thought of "duplicate" `multi-user.target` would actually cause the `google-startup-script` service to be dropped due to a sequencing issue. Adding `After=mulit-user.target` back in _and_ removing `After=cloud-final.service` (as it's "activated" by the "Wants=" directive anyway) fixes the issue, and also tidies up the script a bit so it's more explicit overall.

Fortunately, the old patch was never SRU-ed so this whole problem is limited to lunar.

[Possible Regressions]

My biggest concern was with possible issues with `cloud-init`, as this new patch removes the "After=" directive from `cloud-final.service`. Fortunately though there is still a `Wants=cloud-final.service` directive, so systemd will still attempt to start it. I also ran CPC's `cloud-init` test suite on an image with this new patch, and all was well.

[Test Case]

When a new version of ``google-guest-agent`` is uploaded to ``-proposed``, the following will happen...

 * an image based on -proposed will be built especially for GCE (by the CPC team) and published to the ``ubuntu-os-cloud-image-proposed`` project
 * the GCE team will be asked to validate that the new package addresses the issues it is expected to address, and that the image passes their internal image validation.
 * Each test image will be launched, and we (CPC) will validate:
 ** the package version
 ** that the correct ssh keys have been imported
 ** that the Google specific services are running successfully

If all the testing indicates that the new package is acceptable, verification should be considered to be done.

[Other Information]

Unlike most `google-guest-agent` bugs submitted, this one does not need an SRU as the issue on exists in lunar. I will attach the new patch for ease, created using the usual ``quilt`` workflow.

[Chloé Smith (kajiya)]

Attached the patch, I also have before-and-after `systemd-analyse blame` and `systemd-analyse plot` logs/graphs in case more evidence is needed :)

[Chloé Smith (kajiya)]

MP created upstream, just need @utkarsh to approve now

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0005-make-service-directive-explicit.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Chloé Smith (kajiya)]

I can confirm that the new package version in `-proposed` (20220622.00-0ubuntu5) fixes the issue:

The following test cases had the package version installed:
  ````
  google-guest-agent 20220622.00-0ubuntu5
  ````

* I built a new GCE image using the `-proposed` package and ran CPC's cloud-test-framework
  against it (image name: `testing-ubuntu-guest-2304-lunar-amd64-v20230213`).
  * All the relevant tests passed (namely `google-startup-script` and `google-startup`).

* Sergio (@sergio-j-cazzolato) from the SnapD team needed a functioning lunar GCE image with
  `startup-script` capability for the work he's doing. I directed him to the newly built image
  (`testing-ubuntu-guest-2304-lunar-amd64-v20230213`) and his feedback was that it all worked for
  him (https://chat.canonical.com/canonical/pl/sdycugfzptnn9pimzywyggxmia)

[Launchpad Janitor (janitor)]

This bug was fixed in the package google-guest-agent - 20220622.00-0ubuntu5

---------------
google-guest-agent (20220622.00-0ubuntu5) lunar; urgency=medium

  * d/p/0005-make-service-directive-explicit.patch: restructure directives
    so services run in the correct order. (LP: #2004606)

 -- Chloé 'kajiya' Smith <email address hidden> Fri, 03 Feb 2023 00:14:03 +0000