Include cloud-id in user-agent string

[Andreas Hasenack]

[Impact]

We would like to include a cloud_id/$name parameter to the user-agent string that is sent to https://motd.ubuntu.com. This will allow the server part of motd.ubuntu.com to serve cloud-specific content if one is available.

[Test Case]
a) confirm that the motd message is still displayed:
$ sudo sh /etc/update-motd.d/50-motd-news --force

 * Kata Containers are now fully integrated in Charmed Kubernetes 1.16!
   Yes, charms take the Krazy out of K8s Kata Kluster Konstruction.

     https://ubuntu.com/kubernetes/docs/release-notes

b) confirm that cloud_id is set in the user-agent. This should be tested in lxd, gce, aws, and azure. Here is an example for lxd:

$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1| grep -E "curl .*cloud_id/"
+ curl --connect-timeout 60 --max-time 60 -A curl/7.47.0-1ubuntu2.14 Ubuntu/16.04.6/LTS GNU/Linux/4.15.0-64-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz uptime/921.00/900.00 cloud_id/lxd -o- https://motd.ubuntu.com

For each of the mentioned clouds, we expect cloud_id to be set to their short names: aws, gce, azure.

For xenial, there is an extra test case:
c) list the timers to confirm motd-news is active, and will run sometime in the next 12h following the installation of the updated package:
$ systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2019-09-30 19:38:55 UTC 6h left n/a n/a motd-news.timer motd-news.service
Mon 2019-09-30 22:45:13 UTC 9h left n/a n/a apt-daily.timer apt-daily.service
Tue 2019-10-01 06:12:11 UTC 16h left n/a n/a apt-daily-upgrade.timer apt-daily-upgrade.service
Tue 2019-10-01 13:11:41 UTC 23h left Mon 2019-09-30 13:11:41 UTC 14min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

[Regression Potential]
This change adds a new parameter to the already existing user-agent string. This relies on the cloud-id tool from cloud-init. Even though it's considered trusted to some degree, it may be possible for some clouds to override their name in the data fed to cloud-init, which is what cloud-id parses. Therefore some precautions were taken when handling its contents:
- set value to unknown if cloud-id fails, doesn't exist, or produce an empty value
- sanitize the output of cloud-id, and truncate it to a maximum length
- use a tempfile to store the intermediate output of cloud-id instead of a shell variable. One reason being avoiding storing a potentially large amount of data in a shell variable, and the other is to avoid mistakes when manipulating a shell variable that could contain shell metacharacters.

[Other Info]
For xenial, this is also finishing the fix for bug #1637800 which was stuck in the SRU queue due to it failing checks. Since the xenial package isn't using debhelper, manual dh snippets were added to the maintainer scripts, but they missed the dh_systemd_enable chunks. I'm also skipping starting motd-news.service, since the timer unit will handle that. This avoids an annoying error message about trying to start a unit that is not enabled.

[Original Description]
I'm preemptively filing this bug in case this isn't uploaded before the feature freeze.

We would like to include a cloud_id/$name parameter to the user-agent string that is sent to https://motd.ubuntu.com. This will allow the server part of motd.ubuntu.com to serve cloud-specific content if one is available.

There is an MP for this already at https://code.launchpad.net/~ahasenack/ubuntu/+source/base-files/+git/base-files/+merge/371370

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.2ubuntu5

---------------
base-files (10.2ubuntu5) eoan; urgency=medium

  * motd/50-motd-news: add cloud_id to user-agent string (LP: #1840946)

 -- Andreas Hasenack <email address hidden> Thu, 15 Aug 2019 14:57:39 -0300

[Łukasz Zemczak (sil2100)]

Generally I'd say this is more of a feature than a bug-fix, so a bit tricky from an SRU POV. But on the other hand, this does seem like something we'd like to have in-sync for all stable series. The test case is sufficiently well formed and regression potential thoroughly laid out - this feels solid enough as far as acceptance criteria go.

[Łukasz Zemczak (sil2100)]

Hello Andreas, or anyone else affected,

Accepted base-files into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/10.1ubuntu9.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Andreas, or anyone else affected,

Accepted base-files into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/10.1ubuntu2.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Andreas, or anyone else affected,

Accepted base-files into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/9.4ubuntu4.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

The verification log is quite extensive and verbose, since it's about 3 ubuntu releases and on 4 clouds: lxd, aws, gce, azure. I'll post a summary and an attachment with the detailed log for each.

[Andreas Hasenack (ahasenack)]

Disco verification: passed

Summary of tests:
a) motd message still displayed
b) cloud_id set to $cloudname in user-agent

Tested on: lxd, aws, gce, azure

Using package base-files 10.1ubuntu9.2 from disco-proposed

Detailed test logs attached.

[Andreas Hasenack (ahasenack)]

Bionic verification: passed

Summary of tests:
a) motd message still displayed
b) cloud_id set to $cloudname in user-agent

Tested on: lxd, aws, gce, azure

Using package base-files 10.1ubuntu2.7 from bionic-proposed

Detailed test logs attached.

[Andreas Hasenack (ahasenack)]

Xenial verification: passed

Summary of tests:
No errors or warnings during package upgrade
a) motd message is now displayed
a.1) motd message displayed on login
b) cloud_id set to $cloudname in user-agent
c) motd-news.timer scheduled to run

Tested on: lxd, aws, gce, azure

Using package base-files 9.4ubuntu4.11 from xenial-proposed

Detailed test logs attached.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.1ubuntu9.2

---------------
base-files (10.1ubuntu9.2) disco; urgency=medium

  * update-motd.d/50-motd-news: add cloud_id to user-agent string
    (LP: #1840946)

 -- Andreas Hasenack <email address hidden> Fri, 27 Sep 2019 15:23:40 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for base-files has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.1ubuntu2.7

---------------
base-files (10.1ubuntu2.7) bionic; urgency=medium

  * update-motd.d/50-motd-news: add cloud_id to user-agent string
    (LP: #1840946)

 -- Andreas Hasenack <email address hidden> Fri, 27 Sep 2019 15:24:34 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 9.4ubuntu4.11

---------------
base-files (9.4ubuntu4.11) xenial; urgency=medium

  * update-motd.d/50-motd-news: add cloud_id to user-agent string
    (LP: #1840946)
  * d/postinst.in, d/postrm: add missing dh_systemd_enable pieces. This
    completes the fix for LP #1637800
  * d/postinst.in: do not start the motd-news.service, as it won't be
    in an enabled state anyway, and the timer will run it when needed. This
    addresses the motd-news.service bit in comment #19 of #1637800.

base-files (9.4ubuntu4.10) xenial; urgency=medium

  * Fix syntax error in new base-files postinst code. LP: #1836236.

base-files (9.4ubuntu4.9) xenial; urgency=medium

  * Backport /etc/update-motd.d/50-motd-news, along with the systemd units
    that are used to keep the local cache file up-to-date. LP: #1637800

 -- Andreas Hasenack <email address hidden> Mon, 30 Sep 2019 10:31:01 -0300