pro sometimes runs before cloud-config.service

[Lucas Albuquerque Medeiros de Moura]

[ Impact ]
Currently, the Pro client support a daemon named ubuntu-advantage.service that
performs two actions:

* Actively look for Pro licenses on Azure and GCP images to perform an auto-attach
* Retry auto-attach on Pro images if that command fails on boot

Therefore, this daemon is only being activated on generic Azure and GCP images and all Pro cloud images.

This daemon was originally setup to run after the cloud-config.service. However,
due to a race condition, this is no longer happening. Right now, we manually
check in the daemon code to see if the cloud-config service has finished.

Unfortunately, this new logic now breaks the current Pro setup through cloud-init userdata in both GCP and Azure Pro cloud images. That is because our daemon is now running before cloud-init has even started running. This means that the daemon will perform the attach and not cloud-init itself. This will be clearer, in the following example:

Let's imagine this situation where a user is launching a Pro GCP image:

1) User provides the following cloud-init userdata to the cloud image before booting it:

#cloud-config

ubuntu_advantage:
  enable: []

This means that the user wants no services to be enabled, but still want to attach to the Pro license.

2) Our daemon starts running before cloud-config.service has even started
3) Our daemon see the cloud-config.service as inactive and proceeds normally
4) Our daemon identifies that the user is running on a GCP instance and there is a valid Pro license for it.
5) Due to that, our daemon auto-attach the machine completely ignoring the cloud-init directives.

Therefore, to fix that issue we need to guarantee that we will only execute the daemon, if and only if, cloud-init has already started. That is because, on this situation, the cloud-config.service will already perform the attach operation following the user directives. When the daemon starts running, it will see that the image is already attached and do nothing.

Finally, given this scenario, this bug is only affecting GCP/Azure Pro images, as these are the only ones that will be able to reach the flow described here.

[Discussion]

To address that issue, we are now also checking if the cloud-init service
has already started if we detect that cloud-config service is inactive. If it isn't, the daemon will sleep for an specific amount of time before trying again.

[ Test Plan ]
Since this is a first boot issue, we will need to create a custom image with the package in proposed. Then, we need to guarantee that Pro configuration delivered
through cloud-init is being honored when we launch the image.

Additionally, it is worth noting that we cannot reproduce this issue on a VM easily. That is because, we would need "mock" the VM to pass as one of the affected clouds and also add a valid Pro license to it.

Build image that pulls pro from -proposed but otherwise follows the standard pro image build hook. Upload and register the image with the cloud for testing.

#Set cloud-init userdata that disables all pro services
$ cat userdata.yaml
#cloud-config

ubuntu_advantage:
  enable: []

#Instantiate VM (GCP)
$ gcloud compute instances create pro-order-bug-mantic --image [IMAGE_NAME] --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

#Instantiate VM (Azure)
[TODO]

#On VM, validate version of pro and bugfix (services disable, no cloud-init warnings in log)
$ apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 31.2.3~[RELEASE]
  Candidate: 31.2.3~[RELEASE]

$ cat /var/log/cloud-init.log | grep 'WARNING'

$ pro status
SERVICE AVAILABLE DESCRIPTION
anbox-cloud yes Scalable Android in the cloud
esm-apps yes Expanded Security Maintenance for Applications
esm-infra yes Expanded Security Maintenance for Infrastructure
landscape yes Management and administration tool for Ubuntu
livepatch yes Current kernel is not supported

For a list of all Ubuntu Pro services, run 'pro status --all'

This machine is not attached to an Ubuntu Pro subscription.
See https://ubuntu.com/pro

Supported livepatch kernels are listed here: https://ubuntu.com/security/livepatch/docs/kernels

If the bug is still present, there will be a WARNING in the cloud-init log and pro status will return something similar to:
SERVICE ENTITLED STATUS DESCRIPTION
anbox-cloud yes disabled Scalable Android in the cloud
esm-apps yes enabled Expanded Security Maintenance for Applications
esm-infra yes enabled Expanded Security Maintenance for Infrastructure
fips-preview yes disabled Preview of FIPS crypto packages undergoing certification with NIST
fips-updates yes disabled FIPS compliant crypto packages with stable security updates
livepatch yes enabled Canonical Livepatch service
usg yes disabled Security compliance and audit tools

For a list of all Ubuntu Pro services, run 'pro status --all'
Enable services with: pro enable <service>

                Account: ubuntu-catred
           Subscription: ubuntu-catred
            Valid until: Fri Dec 31 00:00:00 9999 UTC
Technical support level: essential

[ Where problems could occur ]
We are updating the cloud-init wait logic on the daemon. This could potentially make our daemon to not start. However, since we are just now waiting on the base cloud-init.service to start and we have already tested this solution in a custom image, we believe this is a low risk for this fix.

[ Original Description ]
We have recently updated the Pro to not strictly run after cloud-config.service. If cloud-config.service has not been started when pro runs, it can complete before cloud-config.service begins and thus the user-specificed pro configuration will be ignored since the instance is already attached.

When cloud-config.service has yet to run, ubuntu-advantage.service should wait until it's finished before running.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 31.2.3

---------------
ubuntu-advantage-tools (31.2.3) noble; urgency=medium

  * daemon: wait for cloud-init.service to fully activate (LP: #2059952)

 -- Lucas Moura <email address hidden> Tue, 02 Apr 2024 10:13:32 -0300

[Andreas Hasenack (ahasenack)]

> However, CPC is already aware of this issue and will help us creating the test plan here.

I understand CPC can hit this bug quite frequently in the affected cloud images, so releasing this SRU to updates is conditional on an actual test plan being better described by them, and followed.

[Andreas Hasenack (ahasenack)]

Hello Lucas, or anyone else affected,

Accepted ubuntu-advantage-tools into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/31.2.3~23.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Lucas, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/31.2.3~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Lucas, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/31.2.3~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Lucas, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/31.2.3~18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Lucas, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/31.2.3~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Catherine Redfield (catred)]

Test plan details:

Build image that pulls pro from -proposed. If necessary, I can expand on the exact bartender command/changes made. Upload and register the image with a cloud (GCP will be used for testing since that was where I first observed the bug and could reliably reproduce).

#Instantiate VM
$ cat userdata.yaml
#cloud-config

ubuntu_advantage:
  enable: []
$ gcloud compute instances create pro-order-bug-mantic --image [IMAGE_NAME] --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

#On VM, validate version of pro and bugfix (services disable, no cloud-init warnings in log)
$ apt-cache policy ubuntu-pro-client
$ cat /var/log/cloud-init.log | grep 'WARNING'
$ pro status

[Andreas Hasenack (ahasenack)]

Looks good, but please also include:
- azure run (to at least show it's still working, if you cannot reproduce the bug there easily)
- what a test failure looks like, according to this plan

+1 to go ahead and update the bug description's [test plan].

[Catherine Redfield (catred)]

Jammy Validation GCP:

$ gcloud compute instances create pro-order-bug --image testing-ubuntu-2204-jammy-v20240416 --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 31.2.3~22.04
  Candidate: 31.2.3~22.04
  Version table:
 *** 31.2.3~22.04 100
        100 /var/lib/dpkg/status
     31.2.2~22.04 500 (phased 40%)
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING'
catred@pro-order-bug:~$ pro status
SERVICE ENTITLED STATUS DESCRIPTION
anbox-cloud yes disabled Scalable Android in the cloud
esm-apps yes disabled Expanded Security Maintenance for Applications
esm-infra yes disabled Expanded Security Maintenance for Infrastructure
fips-preview yes disabled Preview of FIPS crypto packages undergoing certification with NIST
fips-updates yes disabled FIPS compliant crypto packages with stable security updates
livepatch yes disabled Canonical Livepatch service
usg yes disabled Security compliance and audit tools

For a list of all Ubuntu Pro services, run 'pro status --all'
Enable services with: pro enable <service>

                Account: ubuntu-catred
           Subscription: ubuntu-catred
            Valid until: Fri Dec 31 00:00:00 9999 UTC
Technical support level: essential

No warnings from cloud-init; all services are disabled.

[Catherine Redfield (catred)]

Focal Validation GCP:

$ gcloud compute instances create pro-order-bug --image testing-ubuntu-2004-focal-v20240416 --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 31.2.3~20.04
  Candidate: 31.2.3~20.04
  Version table:
 *** 31.2.3~20.04 100
        100 /var/lib/dpkg/status
     31.2.2~20.04 500
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING'
catred@pro-order-bug:~$ pro status
SERVICE ENTITLED STATUS DESCRIPTION
anbox-cloud yes disabled Scalable Android in the cloud
esm-apps yes disabled Expanded Security Maintenance for Applications
esm-infra yes disabled Expanded Security Maintenance for Infrastructure
fips yes disabled NIST-certified FIPS crypto packages
fips-updates yes disabled FIPS compliant crypto packages with stable security updates
livepatch yes disabled Canonical Livepatch service
usg yes disabled Security compliance and audit tools

For a list of all Ubuntu Pro services, run 'pro status --all'
Enable services with: pro enable <service>

                Account: ubuntu-catred
           Subscription: ubuntu-catred
            Valid until: Fri Dec 31 00:00:00 9999 UTC
Technical support level: essential

No warnings from cloud-init; all services are disabled.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (31.2.3~22.04) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

update-manager/1:22.04.20 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Catherine Redfield (catred)]

Bionic Validation GCP:

$ gcloud compute instances create pro-order-bug --image testing-ubuntu-1804-bionic-v20240417 --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 31.2.3~18.04
  Candidate: 31.2.3~18.04
  Version table:
 *** 31.2.3~18.04 100
        100 /var/lib/dpkg/status
     31.2.2~18.04 500
        500 http://us-central1.gce.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING'
catred@pro-order-bug:~$ pro status
SERVICE ENTITLED STATUS DESCRIPTION
cc-eal yes disabled Common Criteria EAL2 Provisioning Packages
cis yes disabled Security compliance and audit tools
esm-apps yes disabled Expanded Security Maintenance for Applications
esm-infra yes disabled Expanded Security Maintenance for Infrastructure
fips yes disabled NIST-certified FIPS crypto packages
fips-updates yes disabled FIPS compliant crypto packages with stable security updates
livepatch yes disabled Canonical Livepatch service

For a list of all Ubuntu Pro services, run 'pro status --all'
Enable services with: pro enable <service>

                Account: ubuntu-catred
           Subscription: ubuntu-catred
            Valid until: Fri Dec 31 00:00:00 9999 UTC
Technical support level: essential

No warnings from cloud-init; all services are disabled.

[Catherine Redfield (catred)]

Xenial Validation GCP:

$ gcloud compute instances create pro-order-bug --image testing-ubuntu-1604-xenial-v20240417 --image-project ubuntu-catred --metadata-from-file=user-data=userdata.yaml --zone us-central1-a

catred@pro-order-bug:~$ apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 31.2.3~16.04
  Candidate: 31.2.3~16.04
  Version table:
 *** 31.2.3~16.04 100
        100 /var/lib/dpkg/status
     31.2.2~16.04 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
catred@pro-order-bug:~$ cat /var/log/cloud-init.log | grep 'WARNING'
catred@pro-order-bug:~$ pro status
SERVICE AVAILABLE DESCRIPTION
cc-eal yes Common Criteria EAL2 Provisioning Packages
cis yes Security compliance and audit tools
esm-apps yes Expanded Security Maintenance for Applications
esm-infra yes Expanded Security Maintenance for Infrastructure
fips yes NIST-certified FIPS crypto packages
fips-updates yes FIPS compliant crypto packages with stable security updates
livepatch yes Current kernel is not supported
ros yes Security Updates for the Robot Operating System
ros-updates yes All Updates for the Robot Operating System
NOTICES
Operation in progress: pro.daemon.attempt_auto_attach

For a list of all Ubuntu Pro services, run 'pro status --all'

This machine is not attached to an Ubuntu Pro subscription.
See https://ubuntu.com/pro

Supported livepatch kernels are listed here: https://ubuntu.com/security/livepatch/docs/kernels

No warnings from cloud-init; all services are disabled.

[Catherine Redfield (catred)]

We do not publish GCP pro images for mantic so the bug does occur there, hence there is no validation for mantic GCP.