deprecate ubuntu_advantage config key from user-data in favor of ubuntu_pro

[Chad Smith]

[ Impact ]

 * This backport supplements cloud-config user-data schema to allow customers to provide an `ubuntu_pro:` key in cloud-config user-data to auto-attach non-pro images to Ubuntu Pro at instance launch. This ubuntu_pro key support is provided in addition to the existing `ubuntu-advantage:` keys that are currently supported on Xenial/Bionic. This allows uniform configuration and tooling when launching instances across either ESM releases and active supported LTS such as Jammy and Noble.

* It also standardizes customer documentation for 'how to attach to Ubuntu Pro' to allow a single documented procedure and config that is applicable across ESM and active support LTS. It provides customers with the ability to launch any instance in Xenial and Bionic and automatically attach or modify Ubuntu Pro services using the same #cloud-config user-data which can also be used on all active supported LTSes such as Jammy and Noble.
* There are additional motivations to migrate away from ubuntu-advantage and ubuntu_advantage references in all Canonical products as the product name Ubuntu Advantage has been replaced with Ubuntu Pro, so aligning customer expectation with the proper product name reduces customer confusion and documentation concerns.

* Without this backport to unify cloud-config schema for Pro-related functionality, users wishing to attach Xenial, Bionic and Jammy/Noble VMs with cloud-config user-data have to provide separate user-data for each release. Like the following:

- Xenial or Bionic token attach and limit services enabled to esm-infra

#cloud-config
ubuntu-advantage:
   token: <your_token>
   enable: [esm-infra]

-- or --

#cloud-config
ubuntu_advantage:
   token: <your_token>
   enable: [esm-infra]

- Focal, Jammy and Noble
#cloud-config
ubuntu_pro:
   token: <your_token>
    enable: [esm-infra]

* This uploaded fixes this feature gap by addition supplemental cloud-config schema support for ubuntu_pro: key while also retaining support for previous ubuntu-advantage and ubuntu_advantage schema.

[ Test Plan ]

SRU verification manual testing to perform:

- SRU behavior verification:

1. Assert ubuntu_pro key is ignored on released xenial cloud-init
2. Assert ubuntu_pro key is honored on -proposed cloud-init. (pro detach/cloud-init clean)
3. Assert ubuntu-advantage key is honored on -proposed cloud-init (pro detach/cloud-init clean)
4. Assert ubuntu-advantage key debug logs about deprecation in /var/log/cloud-init.og

```
#!/bin/bash
set -ex
RELEASE=$1
UA_TOKEN=$2
VM_NAME=backport-$1-ubuntu-pro
lxc init ubuntu-daily:$RELEASE --vm $VM_NAME
lxc config device add $VM_NAME config disk source=cloud-init:config

WAIT_CMD="cloud-init status --wait --long"
wait_for_cloud_init() {
sleep 5
while ! lxc exec $VM_NAME -- $WAIT_CMD; do
 echo 'waiting...'
 sleep 5
done
}

if [ $RELEASE = "xenial" ]; then
lxc config set $VM_NAME cloud-init.user-data - << EOF
#cloud-config
packages:
  - ubuntu-advantage-tools
  - linux-image-virtual-hwe-16.04 # 16.04 GA kernel as a problem with vsock
runcmd:
  - mount -t 9p config /mnt
  - cd /mnt
  - ./install.sh
  - cd /
  - umount /mnt
  - systemctl start lxd-agent # XXX: causes a reboot
EOF
else
lxc config set $VM_NAME cloud-init.user-data - << EOF
#cloud-config
packages:
  - ubuntu-advantage-tools
runcmd:
  - mount -t 9p config /mnt
  - cd /mnt
  - ./install.sh
  - cd /
  - umount /mnt
  - systemctl start lxd-agent # XXX: causes a reboot
EOF
fi

lxc start --console $VM_NAME || true # hit enter on prompt

echo "-- confirm cloud-init boot complete"
wait_for_cloud_init

echo "-- setup ubuntu_pro attach user-data"
cat > ubuntu_pro.yaml << EOF
#cloud-config
ubuntu_pro:
  token: $UA_TOKEN
EOF

echo "-- setup ubuntu-advantage attach user-data"
cat > ubuntu_adv.yaml << EOF
#cloud-config
ubuntu-advantage:
  token: $UA_TOKEN
EOF

echo "-- setup ubuntu-advantage commands user-data"
cat > ubuntu_adv_cmd.yaml << EOF
#cloud-config
ubuntu-advantage:
  commands:
     00: [attach, $UA_TOKEN]
EOF

lxc config set $VM_NAME cloud-init.user-data="$(cat ubuntu_pro.yaml)"
lxc exec $VM_NAME -- cloud-init clean --logs --reboot || true

echo "-- confirm orig cloud-init ignores ubuntu_pro"
wait_for_cloud_init
lxc exec $VM_NAME -- pro status --format=json | jq .attached | grep "false" && echo "SUCCESS: expected ignored ubuntu_pro before upgrade" || echo "FAILURE: unexpected pro already attached"

echo "-- Proposed cloud-init honors ubuntu_pro key"
lxc exec $VM_NAME -- add-apt-repository ppa:chad.smith/esm-backport-ubuntu-pro -y
lxc exec $VM_NAME -- apt-get update -y
lxc exec $VM_NAME -- apt-get install cloud-init -y
wait_for_cloud_init
lxc exec $VM_NAME -- cloud-init clean --logs --reboot || true
wait_for_cloud_init
lxc exec $VM_NAME -- pro status --format=json | jq .attached | grep "true" && echo "SUCCESS:ubuntu_pro key honored to properly attach after upgrade" || echo "FAILURE: pro did not properly attach with ubuntu_pro key"

echo "-- Proposed cloud-init still honors ubuntu-advantage key"
lxc exec $VM_NAME -- pro detach --assume-yes
lxc exec $VM_NAME -- rm -rf /var/lib/ubuntu-advantage
sed -i 's/ubuntu_pro/ubuntu-advantage/' ubuntu_pro.yaml
lxc config set $VM_NAME cloud-init.user-data="$(cat ubuntu_adv.yaml)"

lxc exec $VM_NAME -- cloud-init clean --logs --reboot || true
sleep 10
wait_for_cloud_init
lxc exec $VM_NAME -- pro status --format=json | jq .attached | grep "true" && echo "SUCCESS: ubuntu pro properly attached with ubuntu-advantage key after upgrade" || echo "FAILURE: pro did not properly attach with ubuntu-advantage key"
echo "-- Proposed cloud-init logs debug deprecation messages"
lxc exec $VM_NAME -- grep ubuntu-adv /var/log/cloud-init.log

if [ $RELEASE = "xenial" ]; then
echo "-- Proposed cloud-init still honors ubuntu-advantage:commands key on Xenial only"
lxc exec $VM_NAME -- pro detach --assume-yes
lxc exec $VM_NAME -- rm -rf /var/lib/ubuntu-advantage
sed -i 's/ubuntu_pro/ubuntu-advantage/' ubuntu_pro.yaml
lxc config set $VM_NAME cloud-init.user-data="$(cat ubuntu_adv_cmd.yaml)"

lxc exec $VM_NAME -- cloud-init clean --logs --reboot || true
sleep 10
wait_for_cloud_init
lxc exec $VM_NAME -- pro status --format=json | jq .attached | grep "true" && echo "SUCCESS: ubuntu pro properly attached with ubuntu-advantage:commands after upgrade" || echo "FAILURE: pro did not properly attach with ubuntu-advantage:commands"
echo "-- Proposed cloud-init logs debug deprecation messages"
lxc exec $VM_NAME -- grep ubuntu-adv /var/log/cloud-init.log
fi

[ Where problems could occur ]

 * This changeset is limited to the cc_ubuntu_pro(formerly cc_ubuntu_advantage) module and will only potentially impact users who provide #cloud-config user-data containing config keys for ubuntu-advantage, ubuntu_advantage or ubuntu_pro. The cc_ubuntu_pro module itself is not run unless the specific configuration keys are provided in user-data or in the image in /etc/cloud/cloud.cfg.d/*.cfg files. On typical system boots, this module is skipped at the beginning of the cc_ubuntu_pro.handle function.

* Problems could occur in the rename of cc_ubuntu_advantage -> cc_ubunt_pro module if there are customized /etc/cloud/cloud.cfg files in an image. If the customized /etc/cloud/cloud.cfg file retained the former name ubuntu_advantage, and the image maintainer didn't accept upstream packaging change contained here to rename ubuntu_advantage -> ubuntu_pro, the cloud-init would skip running the ubuntu_pro module during boot. This would only affect custom images launched with cloud-config user-data containing pro token attach config and also retaninig an unsupported /etc/cloud/cloud.cfg file in their images. The result would be that cloud-init would succeed booting , but ignore pro-specific configuration from user-data.

[ Other Info ]

[ Original description ]

In upstream cloud-init ubuntu_advantage user-data cloud-config key is deprecated in favor of ubuntu_pro to better align with current Ubuntu Pro product naming and to avoid confusion in howtos, tutorials or tooling when interacting with Ubuntu Pro offerings.

In an effort to standardize product naming across all Ubuntu supported releases, we will target a backport of this key deprecation to Xenial and Bionic to ensure Ubuntu Pro's releases covered by Extended Security Maintenance will also align with active Ubuntu Long Term Support(LTS) and interim release configuration keys.

This bug is an SRU process bug representing the intent to backport a limited scope of functionality related to the cc_ubuntu_advantage module for better Ubuntu Pro support ESM releases.

No other functional changes are intended beyond logged messages, user-data schema and config-key deprecation.

[Robie Basak (racb)]

Isn't this missing SRU documentation? How do you propose to test this change - for example that use of an "old" cloud-config still works as expected?

Is it correct to treat this as a "deprecation" in Bionic, given that what cloud-init originally supported in Bionic should remain supported until Bionic EOLs? The matter of "deprecation" seems separate to me vs. adding an ubuntu_pro key for unification across all releases, which makes sense to do.

[Chad Smith (chad.smith)]

@racb this was missing SRU documentation as far as the previous bionic upload was concerned as we thought originally this upload may be taken direct into esm-infra pockets.

But, because this supplemental cloud-config user-data key is required at initial instance launch in non-pro images, providing a package update in esm-infra doesn't provide non-pro images with the ability to use the unified `ubuntu_pro:` key at initial launch.

I have since added the full SRU template to describe the needs as verification steps to establish that both old config key and new config key are allowed to ensure no regressing for users trying to launch and attach non-pro images to Ubuntu Pro.

Per your deprecation comment in bionic, the deprecation treatment in bionic is only a debug level log that leaves a breadcrumb to inform that you probably shouldn't be using the ubuntu-advantage or ubuntu_advantage keys long term as upgrades to the next stable release may have dropped that functionality in the more recent LTS.

Generally deprecated behavior is not ever something cloud-init will drop across and SRU boundary. We will continue to retain original behavior where ever possible. But, by announcing that it is a deprecated feature, someone who may be thinking of upgrading to a newer release of Ubuntu such as Focal++ can be better aware that this deprecated behavior may cause warnings or be dropped across that Ubuntu release upgrade boundary.

To reinforce this same type of debug level behavior in other active stable releases (focal ++), we have also recently added a feature flag in cloud-init on stable releases to provide this same mechanism of only logging debug level messages for deprecated features[1] which allows each stable downstream release to declare the cloud-init package version boundary at which each deprecated message is considered a deprecation warning level log versus just a debug level log.

References:
[1] https://github.com/canonical/cloud-init/blob/ubuntu/jammy/cloudinit/features.py#L90-L122

[Andreas Hasenack (ahasenack)]

Initially I thought it was only about adding some sort of "alias", such that `ubuntu_pro` would also be recognized alongside `ubuntu_advantage`. But when looking at the diff, I see that it's actually replacing one cloud-init module with another. I see this was also hinted at in the "where problems could occurr" section of the bug description.

My concerns with this approach are as follows:

a) the test plan doesn't seem to cover all other changes brought in via the new ubuntu_pro module. Not only new features, but also that existing ones don't regress behavior if the module is invoked via the old key `ubuntu_advantage`.
b) this doesn't feel like a minimal fix anymore
c) on the justification part, it was said that users should be able to use the same `ubuntu_pro` module configuration for x/b and later releases. But cloud-init SRUs for x/b are no longer being done. What if focal, for example, fixes a bug in the ubuntu_pro module there, would that also be backported to x/b from now on?
d) since this is replacing one module with another, it's very difficult to review the change. What else is the ubuntu_pro module bringing down? How can we be sure it's not changing behavior, specially if being invoked by the (now deprecated) `ubuntu_advantage` key?
e) this x/b SRU will require new images to be built for those releases, outside of ESM. In internal conversations, that seems it's doable, but has there been a formal agreement that it will be done for this SRU? And, considering point (c) above, what about future SRUs?

Is the "users can now use the exact same cloud-init config all the way from x to noble" really true, or would it just apply to Pro configuration? Note that bionic and xenial have very different cloud-init versions:

xenial: 21.1-19-gbad84ad4
bionic: 23.1.2
focal and later: 24.2 (with 24.3.1 in proposed)

Considering all of the above, do you think it's still worth to pursue this SRU?

[Timo Aaltonen (tjaalton)]

@chad, please see the comments by andreas

[Timo Aaltonen (tjaalton)]

@chad, ping?