Bug #1888572 “motd-news: use wget instead of curl” : Bugs : base-files package : Ubuntu

Bug Description

[Impact]
The motd-news script is using curl, but since that is an optional package, there is no guarantee that it will be installed. The script correctly checks for its presence before trying to use it, though, so it won't fail. As we don't want to add such a dependency to the base-files package, we should switch to wget, which is standard.

[Test Case]
wget has a different behavior than curl in some areas, one of which is crucial for the motd-per-cloud feature. While curl will only complain about a 404 from the server if given a specific parameter (-f), wget does that by default, and needs special handling.

a) With curl, base-files and ubuntu-server installed, first verify motd-news works:
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

Now remove curl, and retry. It should exit immediately with no output:

$ sudo /etc/update-motd.d/50-motd-news --force

Install the updated base-files package and the new motd-news-config package from proposed:

$ sudo apt install base-files motd-news-config

Note curl is still not available:
$ curl

Command 'curl' not found, but can be installed with:

sudo apt install curl

Re-run the motd-news script, this time it should produce output again:
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

b) Verify motd-news per cloud remains working.
If you have /usr/bin/cloud-id, copy it to a backup:
sudo cp /usr/bin/cloud-id{,.orig}

Create a new one, per supported cloud. For aws, for example:
echo -e '#!/bin/sh\necho aws' | sudo tee /usr/bin/cloud-id

Confirm by running it:
$ cloud-id
aws

And confirm motd-news keeps working (it might return different content):
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

Repeat for the gce and azure clouds, by changing the cloud-id script accordingly.

To confirm the right cloud_id is being used, use sh -x and grep for its output:

$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/azure -O- --content-on-error https://motd.ubuntu.com

This also verifies again it's using wget instead of curl.

[Regression Potential]
Possible regressions will likely be tied to a difference in behavior between curl and wget. In fact, one was caught[1] in the development release and the fix is included here, with a test.

[Other Info]
N/A

1. https://bugs.launchpad.net/bugs/1889117

See original description

Tags: Edit Tag help

Related branches

~ahasenack/ubuntu/+source/base-files:focal-handle-was-removed-1895302

Superseded for merging into ubuntu/+source/base-files:ubuntu/devel

Canonical Server Core Reviewers: Pending requested 2020-09-15

Canonical Server Team: Pending requested 2020-09-15

~ahasenack/ubuntu/+source/base-files:xenial-motd-news-config-split

Merged into ubuntu/+source/base-files:ubuntu/xenial-devel at revision c91565c6eb7c49f151ab7fff5a1cb9e350224e04

Christian Ehrhardt : Approve on 2020-08-17

Canonical Server Team: Pending requested 2020-08-13

~ahasenack/ubuntu/+source/base-files:bionic-motd-news-config-split

Merged into ubuntu/+source/base-files:ubuntu/bionic-devel at revision 08744d99f2c353f8db78e777d5f79044c1a1a8c2

Christian Ehrhardt : Approve on 2020-08-17

Canonical Server Team: Pending requested 2020-08-07

~ahasenack/ubuntu/+source/base-files:focal-motd-news-config-split

Merged into ubuntu/+source/base-files:ubuntu/focal-devel at revision 6f3fbdb8ed9b84ed92aee611b03ae0bb96f4b7c4

Christian Ehrhardt : Approve on 2020-08-17

Canonical Server Team: Pending requested 2020-08-06

Andreas Hasenack (ahasenack) wrote on 2020-07-22:

This was fixed for groovy in https://launchpad.net/ubuntu/+source/base-files/11ubuntu9

Changed in base-files (Ubuntu Xenial):
status:	New → In Progress
Changed in base-files (Ubuntu Bionic):
status:	New → In Progress
Changed in base-files (Ubuntu Focal):
status:	New → In Progress
Changed in base-files (Ubuntu Xenial):
assignee:	nobody → Andreas Hasenack (ahasenack)
Changed in base-files (Ubuntu Bionic):
assignee:	nobody → Andreas Hasenack (ahasenack)
Changed in base-files (Ubuntu Focal):
assignee:	nobody → Andreas Hasenack (ahasenack)
Changed in base-files (Ubuntu):
assignee:	Andreas Hasenack (ahasenack) → nobody
status:	In Progress → Fix Released

Andreas Hasenack (ahasenack) on 2020-08-13

description:

updated

Timo Aaltonen (tjaalton) wrote on 2020-08-21: Please test proposed package

Hello Andreas, or anyone else affected,

Accepted base-files into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/11ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in base-files (Ubuntu Focal):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-focal
Changed in base-files (Ubuntu Bionic):
status:	In Progress → Fix Committed
tags:	added: verification-needed-bionic

Timo Aaltonen (tjaalton) wrote on 2020-08-21:

Hello Andreas, or anyone else affected,

Accepted base-files into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/10.1ubuntu2.10 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in base-files (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed-xenial

Timo Aaltonen (tjaalton) wrote on 2020-08-21:

Hello Andreas, or anyone else affected,

Accepted base-files into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/9.4ubuntu4.13 in a few hours, and then in the -proposed repository.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Andreas Hasenack (ahasenack) wrote on 2020-08-25:

Download full text (3.6 KiB)

Focal verification

Test (a)

Current base-files, ubuntu-server and curl installed:
ii base-files 11ubuntu5.1 amd64 Debian base system miscellaneous files
ii curl 7.68.0-1ubuntu2.2 amd64 command line tool for transferring data with URL syntax
ii ubuntu-server 1.450.1 amd64 The Ubuntu Server system

motd-news runs:
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

Without curl:
$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news exits immediately:
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force
ubuntu@focal-motd-news-wget:~$ echo $?
0

With base-files and motd-news-config from focal-proposed:
ubuntu@focal-motd-news-wget:~$ apt-cache policy base-files motd-news-config
base-files:
  Installed: 11ubuntu5.2
  Candidate: 11ubuntu5.2
  Version table:
*** 11ubuntu5.2 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
(...)
motd-news-config:
  Installed: 11ubuntu5.2
  Candidate: 11ubuntu5.2
  Version table:
*** 11ubuntu5.2 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages

Still no curl:
$ dpkg -l curl | grep curl
un curl <none> <none> (no description available)

motd-news works again:
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

Test (b)
Verify motd-news per cloud remains working.

b1) aws
ubuntu@focal-motd-news-wget:~$ cloud-id
aws
$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.

cloud_id/aws was sent:
$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/aws -O- --content-on-error https://motd.ubuntu.com

b2) gce
ubuntu@focal-motd-news-wget:~$ cloud-id
gce
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19? It's nearly here! Try RC3 with
sudo snap install microk8s --channel=1.19/candidate --classic

https://microk8s.io/ has docs and details.
ubuntu@focal-motd-news-wget:~$ sudo sh -x /etc/update-motd.d/50-motd-news --force 2>&1 | grep -wE 'wget .*cloud_id/[a-z]+'
+ wget --timeout 60 -U wget/1.20.3-1ubuntu1 Ubuntu/20.04.1/LTS GNU/Linux/5.4.0-42-generic/x86_64 Intel(R)/Core(TM)/i7-7600U/CPU/@/2.80GHz cloud_id/gce -O- --content-on-error https://motd.ubuntu.com

b3) azure
ubuntu@focal-motd-news-wget:~$ cloud-id
azure
ubuntu@focal-motd-news-wget:~$ sudo /etc/update-motd.d/50-motd-news --force

* Are you ready for Kubernetes 1.19?...

tags:

added: verification-done-xenial
removed: verification-needed-xenial

Lord Baco (lordbaco-deactivatedaccount) wrote on 2020-08-27:

Can you stop launching wget or curl as root? -- Please!

Launching a web crawler/browser such as wget or curl as super user root is a bad design
specially if it runs in background without user consent and knowledge every 12h on all
Ubuntu Desktop and Server based devices worldwide.

It does not following IT Best Practices and is part of a unremovable package aka base-files
https://github.com/curl/curl/issues/5557

REFERENCES
https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/1867424

Mathew Hodson (mhodson) on 2020-08-31

tags:

removed: verification-needed

Launchpad Janitor (janitor) wrote on 2020-09-02:

This bug was fixed in the package base-files - 9.4ubuntu4.13

---------------
base-files (9.4ubuntu4.13) xenial; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/postinst.in, d/postrm, d/preinst: remove /etc/default/motd-news
      config file on base-files upgrade using dpkg-maintscript-helper
    - d/rules: install d/preinst
    - d/control: break on ubuntu-server << 1.361.5 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/conffiles: remove motd-news
    - d/rules, d/motd-news-config.conffiles: packaging motd-news-config
      without debhelper

  [ Steve Langasek ]
  * motd/50-motd-news: use wget instead of curl, since wget is standard but
    curl is optional (LP: #1888572):
    - This changes the timeout behavior slightly because wget does not have
      an exact equivalent to curl's --max-time argument, we are using
      --timeout instead.

-- Andreas Hasenack <email address hidden> Mon, 17 Aug 2020 11:19:19 -0300

Changed in base-files (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Steve Langasek (vorlon) wrote on 2020-09-02: Update Released

#10

The verification of the Stable Release Update for base-files has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote on 2020-09-02:

#11

This bug was fixed in the package base-files - 10.1ubuntu2.10

---------------
base-files (10.1ubuntu2.10) bionic; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/base-files.maintscript: remove /etc/default/motd-news config file
      on upgrade
    - d/control: break on ubuntu-server << 1.417.5 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/rules, d/motd-news-config.install: /e/d/motd-news is in the
      motd-news-config package now

-- Andreas Hasenack <email address hidden> Thu, 13 Aug 2020 15:59:47 -0300

Changed in base-files (Ubuntu Bionic):
status:	Fix Committed → Fix Released

Launchpad Janitor (janitor) wrote on 2020-09-02:

#12

This bug was fixed in the package base-files - 11ubuntu5.2

---------------
base-files (11ubuntu5.2) focal; urgency=medium

  [ Andreas Hasenack ]
  * motd/50-motd-news: don't include uptime in the user-agent string
    (LP: #1886572)
  * Move the /etc/default/motd-news conffile to the motd-news-config
    package (LP: #1888575):
    - d/base-files.maintscript: remove /etc/default/motd-news config file
      on upgrade
    - d/control: break on ubuntu-server << 1.450.2 to force an upgrade if
      it is installed, which will pull motd-news-config and the conffile
      back in
    - d/motd-news-config.postinst:
      + handle the upgrade case where the motd-news config file was
        changed while it belonged to base-files
      + disable motd-news if the config file was removed by hand before
        the upgrade
    - d/postinst.in: signal the motd-news-config package if the
      motd-news config file was removed manually before the upgrade
    - d/control: new motd-news-config package, carrying the
      configuration file for the /etc/update-motd.d/50-motd-news script.
    - d/rules, d/motd-news-config.install: /e/d/motd-news is in the
      motd-news-config package now

-- Andreas Hasenack <email address hidden> Mon, 17 Aug 2020 10:31:58 -0300

Changed in base-files (Ubuntu Focal):
status:	Fix Committed → Fix Released

Ubuntu
base-files package

motd-news: use wget instead of curl

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntubase-files package

motd-news: use wget instead of curl

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
base-files package