motd-news: use wget instead of curl

[Andreas Hasenack]

[Impact]
The motd-news script is using curl, but since that is an optional package, there is no guarantee that it will be installed. The script correctly checks for its presence before trying to use it, though, so it won't fail. As we don't want to add such a dependency to the base-files package, we should switch to wget, which is standard.

[Test Case]
wget has a different behavior than curl in some areas, one of which is crucial for the motd-per-cloud feature. While curl will only complain about a 404 from the server if given a specific parameter (-f), wget does that by default, and needs special handling.

a) With curl, base-files and ubuntu-server installed, first verify motd-news works:
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Now remove curl, and retry. It should exit immediately with no output:

$ sudo /etc/update-motd.d/50-motd-news --force

Install the updated base-files package and the new motd-news-config package from proposed:

$ sudo apt install base-files motd-news-config

Note curl is still not available:
$ curl

Command 'curl' not found, but can be installed with:

sudo apt install curl

Re-run the motd-news script, this time it should produce output again:
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

b) Verify motd-news per cloud remains working.
If you have /usr/bin/cloud-id, copy it to a backup:
sudo cp /usr/bin/cloud-id{,.orig}

Create a new one, per supported cloud. For aws, for example:
echo -e '#!/bin/sh\necho aws' | sudo tee /usr/bin/cloud-id

Confirm by running it:
$ cloud-id
aws

And confirm motd-news keeps working (it might return different content):
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Repeat for the gce and azure clouds, by changing the cloud-id script accordingly.

To confirm the right cloud_id is being used, use sh -x and grep for its output:

$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/azure -O- --content-on-error https://motd.ubuntu.com

This also verifies again it's using wget instead of curl.

[Regression Potential]
Possible regressions will likely be tied to a difference in behavior between curl and wget. In fact, one was caught[1] in the development release and the fix is included here, with a test.

[Other Info]
N/A

1. https://bugs.launchpad.net/bugs/1889117

[Andreas Hasenack (ahasenack)]

This was fixed for groovy in https://launchpad.net/ubuntu/+source/base-files/11ubuntu9

[Timo Aaltonen (tjaalton)]

Hello Andreas, or anyone else affected,

Accepted base-files into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/11ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Andreas, or anyone else affected,

Accepted base-files into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/10.1ubuntu2.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Andreas, or anyone else affected,

Accepted base-files into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/9.4ubuntu4.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Focal verification

Test (a)

Current base-files, ubuntu-server and curl installed:
ii base-files 11ubuntu5.1 amd64 Debian base system miscellaneous files
ii curl 7.68.0-1ubuntu2.2 amd64 command line tool for transferring data with URL syntax
ii ubuntu-server 1.450.1 amd64 The Ubuntu Server system

motd-news runs:
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Without curl:
$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news exits immediately:
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force
ubuntu@focal-motd-news-wget:~$ echo $?
0

With base-files and motd-news-config from focal-proposed:
ubuntu@focal-motd-news-wget:~$ apt-cache policy base-files motd-news-config
base-files:
  Installed: 11ubuntu5.2
  Candidate: 11ubuntu5.2
  Version table:
 *** 11ubuntu5.2 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
(...)
motd-news-config:
  Installed: 11ubuntu5.2
  Candidate: 11ubuntu5.2
  Version table:
 *** 11ubuntu5.2 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages

Still no curl:
$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news works again:
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Test (b)
Verify motd-news per cloud remains working.

b1) aws
ubuntu@focal-motd-news-wget:~$ cloud-id
aws
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

cloud_id/aws was sent:
$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/aws -O- --content-on-error https://motd.ubuntu.com

b2) gce
ubuntu@focal-motd-news-wget:~$ cloud-id
gce
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@focal-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/gce -O- --content-on-error https://motd.ubuntu.com

b3) azure
ubuntu@focal-motd-news-wget:~$ cloud-id
azure
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19?...

[Andreas Hasenack (ahasenack)]

Bionic verification

Test (a)

Current base-files, ubuntu-server and curl installed:
ii base-files 10.1ubuntu2.9 amd64 Debian base system miscellaneous files
ii curl 7.58.0-2ubuntu3.10 amd64 command line tool for transferring data with URL syntax
ii ubuntu-server 1.417.4 amd64 The Ubuntu Server system

motd-news runs:
$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Without curl, it exits immediately:
ubuntu@bionic-motd-news-wget:~$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)
ubuntu@bionic-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force
ubuntu@bionic-motd-news-wget:~$ echo $?
0
ubuntu@bionic-motd-news-wget:~$

Installing base-files and motd-news-config from bionic-proposed:
ubuntu@bionic-motd-news-wget:~$ apt-cache policy base-files motd-news-config
base-files:
  Installed: 10.1ubuntu2.10
  Candidate: 10.1ubuntu2.10
  Version table:
 *** 10.1ubuntu2.10 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
(...)
motd-news-config:
  Installed: 10.1ubuntu2.10
  Candidate: 10.1ubuntu2.10
  Version table:
 *** 10.1ubuntu2.10 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages

Still no curl:
$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news works again:
ubuntu@bionic-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@bionic-motd-news-wget:~$ echo $?
0

Test (b)
Verify motd-news per cloud remains working.

b1) aws
ubuntu@bionic-motd-news-wget:~$ cloud-id
aws
ubuntu@bionic-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@bionic-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.19.4-1ubuntu2.2 Ubuntu/18.04.5/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/aws -O- --content-on-error https://motd.ubuntu.com

b2) gce
ubuntu@bionic-motd-news-wget:~$ cloud-id
gce
ubuntu@bionic-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@bionic-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.19.4-1ubuntu2.2 Ubuntu/18.04.5/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.8...

[Andreas Hasenack (ahasenack)]

Xenial verification

Test (a)

Current base-files and curl installed (no ubuntu-server out-of-the-box on xenial lxd images):
ii base-files 9.4ubuntu4.12 amd64 Debian base system miscellaneous files
ii curl 7.47.0-1ubuntu2.16 amd64 command line tool for transferring data with URL syntax

motd-news runs:
ubuntu@xenial-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.

Remove curl, and it exits immediatelly:
ubuntu@xenial-motd-news-wget:~$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

ubuntu@xenial-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force
ubuntu@xenial-motd-news-wget:~$ echo $?
0

With base-files and motd-news-config from xenial-proposed:
base-files:
  Installed: 9.4ubuntu4.13
  Candidate: 9.4ubuntu4.13
  Version table:
 *** 9.4ubuntu4.13 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
(...)
motd-news-config:
  Installed: 9.4ubuntu4.13
  Candidate: 9.4ubuntu4.13
  Version table:
 *** 9.4ubuntu4.13 500
        500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages

Still with curl removed:
ubuntu@xenial-motd-news-wget:~$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news works again:
ubuntu@xenial-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@xenial-motd-news-wget:~$ echo $?
0

Test (b)
Verify motd-news per cloud remains working.

b1) aws
ubuntu@xenial-motd-news-wget:~$ cloud-id
aws
ubuntu@xenial-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@xenial-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.17.1-1ubuntu1.5 Ubuntu/16.04.7/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/aws -O- --content-on-error https://motd.ubuntu.com

b2) gce
ubuntu@xenial-motd-news-wget:~$ cloud-id
gce
ubuntu@xenial-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

 * Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
   sudo snap install microk8s --channel=1.19/candidate --classic

   https://microk8s.io/ has docs and details.
ubuntu@xenial-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.17.1-1ubuntu1.5 Ubuntu/16.04.7/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/gce -O- --content-on-error https://motd.ubuntu.com

b3) azure
ubuntu@...

[B. (b-deactivatedaccount-deactivatedaccount)]

Can you stop launching wget or curl as root? -- Please!

Launching a web crawler/browser such as wget or curl as super user root is a bad design
specially if it runs in background without user consent and knowledge every 12h on all
Ubuntu Desktop and Server based devices worldwide.

It does not following IT Best Practices and is part of a unremovable package aka base-files
https://github.com/curl/curl/issues/5557

REFERENCES
https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1867424

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 9.4ubuntu4.13

---------------
base-files (9.4ubuntu4.13) xenial; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/postinst.in, d/postrm, d/preinst: remove /etc/default/motd-news
      config file on base-files upgrade using dpkg-maintscript-helper
    - d/rules: install d/preinst
    - d/control: break on ubuntu-server << 1.361.5 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/conffiles: remove motd-news
    - d/rules, d/motd-news-config.conffiles: packaging motd-news-config
      without debhelper

  [ Steve Langasek ]
  * motd/50-motd-news: use wget instead of curl, since wget is standard but
    curl is optional (LP: #1888572):
    - This changes the timeout behavior slightly because wget does not have
      an exact equivalent to curl's --max-time argument, we are using
      --timeout instead.

 -- Andreas Hasenack <email address hidden> Mon, 17 Aug 2020 11:19:19 -0300

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for base-files has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 10.1ubuntu2.10

---------------
base-files (10.1ubuntu2.10) bionic; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/base-files.maintscript: remove /etc/default/motd-news config file
      on upgrade
    - d/control: break on ubuntu-server << 1.417.5 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/rules, d/motd-news-config.install: /e/d/motd-news is in the
      motd-news-config package now

  [ Steve Langasek ]
  * motd/50-motd-news: use wget instead of curl, since wget is standard but
    curl is optional (LP: #1888572):
    - This changes the timeout behavior slightly because wget does not have
      an exact equivalent to curl's --max-time argument, we are using
      --timeout instead.

 -- Andreas Hasenack <email address hidden> Thu, 13 Aug 2020 15:59:47 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-files - 11ubuntu5.2

---------------
base-files (11ubuntu5.2) focal; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/base-files.maintscript: remove /etc/default/motd-news config file
      on upgrade
    - d/control: break on ubuntu-server << 1.450.2 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/rules, d/motd-news-config.install: /e/d/motd-news is in the
      motd-news-config package now

  [ Steve Langasek ]
  * motd/50-motd-news: use wget instead of curl, since wget is standard but
    curl is optional (LP: #1888572):
    - This changes the timeout behavior slightly because wget does not have
      an exact equivalent to curl's --max-time argument, we are using
      --timeout instead.

 -- Andreas Hasenack <email address hidden> Mon, 17 Aug 2020 10:31:58 -0300