[SRU] 50-motd-news costs 5 seconds every login on firewalled systems

Bug #1691901 reported by Scott Moser on 2017-05-19
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
base-files (Ubuntu)
High
Dustin Kirkland 
Zesty
High
Dustin Kirkland 
Artful
High
Dustin Kirkland 

Bug Description

I noticed quite a long time to login to some of my systems via ssh (or scp).
Investigating lead me to find out that the '50-motd-news' file
(/etc/update-motd.d/50-motd-news) was the primary cost.

In line 108 [1] it does something like:
 curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com

The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com.
The way the lab is configured, they just end up timing out. So every scp or ssh connection
or other path to trigger update-motd will cost 5 seconds.

======= SRU =======

[ IMPACT ]

This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually.

[ TEST CASE ]

You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately.

Failure case:

$ lxc launch ubuntu:17.04 LP1691901
$ lxc exec LP1691901 bash
# ssh-import-id kirkland
# echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts
# rm -f /var/cache/motd-news
# exit
$ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true
real 0m5.333s
user 0m0.016s
sys 0m0.000s

Apply the fix.

$ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true
real 0m0.316s
user 0m0.008s
sys 0m0.008s

[ REGRESSION ]

This is a simple, safe fix with minimal regression potential:

diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news
--- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.000000000 -0600
+++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.000000000 -0500
@@ -51,9 +51,13 @@
 # If we're not forcing an update, and we have a cached motd-news file,
 # then just print it and exit as quickly as possible, for login performance.
 # Note that systemd should keep this cache file up to date, asynchronously
-if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then
- echo
- safe_print $CACHE
+if [ "$FORCED" != "1" ]; then
+ if [ -r $CACHE ]; then
+ echo
+ safe_print $CACHE
+ else
+ : > $CACHE
+ fi
        exit 0
 fi

@@ -111,7 +115,9 @@
                safe_print "$NEWS"
                # Try to update the cache
                safe_print "$NEWS" 2>/dev/null >$CACHE || true
- fi
+ else
+ : > "$CACHE"
+ fi
 done
 rm -f "$NEWS" "$NEWS.err"
 exit 0

--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: base-files 9.6ubuntu13
ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11
Uname: Linux 4.10.0-21-generic x86_64
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
Date: Fri May 19 01:11:30 2017
Ec2AMI: ami-0000004f
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: base-files
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.update-motd.d.50-motd-news: 2017-05-19T01:10:25.572110

Scott Moser (smoser) wrote :
Steve Langasek (vorlon) on 2017-05-23
summary: - 50-motd-news costs 5 seconds every login on disconnected systems
+ 50-motd-news costs 5 seconds every login on firewalled systems

I've adjusted the bug description to match my understanding of the actual problem.

AIUI this does not happen on a disconnected system; if motd.ubuntu.com cannot resolve or returns ECONNREFUSED or EHOSTUNREACH, there should be no 5-second delay. The error only happens on a system which has routes, but has a firewall that drops packets instead of responding to the request. (Or when motd.u.c is actually down.)

So, what you appear to be describing here is a deliberate design tradeoff. Unless you are recommending a different timeout (if so, what value?) or asking for this to be disabled altogether (in which case I will refer you to Dustin), I think the only solution here is for your lab to have suitable firewall+proxy settings.

Scott Moser (smoser) on 2017-06-13
Changed in base-files (Ubuntu):
status: New → Confirmed
importance: Undecided → High
Scott Moser (smoser) wrote :

Steve,
It should not be a requirement to have connectivity to motd.ubuntu.com to have quick login.
Its quite common to have ingress networking like we see in our labs.

I suggest the fix would be better to background the download and to atomically move it the content into place. Then the reader would just display it if it were present.

Then,
 a.) pure failure to reach (ECONNREFUSED or EHOSTUNREACH) would simply show no data in motd.
 b.) hang would be gone and would simply not have data in the motd
 c.) fast success would be shown on this login
 d.) slower success would be shown on next login (possibly stale, which is always going to be a possibility)

Scott Moser (smoser) wrote :

For reference, 2 ways to disable this:
a.) disable motd entirely
sudo sed -i '/^[^#].*pam_motd/s/^/#/' /etc/pam.d/sshd

b.) disable 50-motd-news
sudo chmod -x /etc/update-motd.d/50-motd-news

Dustin Kirkland  (kirkland) wrote :

You can also edit /etc/default/motd-news and:

 a) set ENABLED=0

 b) changed the URL to a custom one that does resolve or work for you

 c) change the time to wait

Dustin Kirkland  (kirkland) wrote :

By design, the cache file, /var/cache/motd-news, should be populated in the background by a systemd timer that runs roughly once a day.

If that file is populated, then there should be zero impact on login.

Perhaps the background cache population should create an empty file, if it can't reach the $URL?

Scott Moser (smoser) wrote :
Changed in base-files (Ubuntu):
status: Confirmed → Triaged
Changed in base-files (Ubuntu):
assignee: nobody → Dustin Kirkland  (kirkland)
status: Triaged → In Progress
Changed in base-files (Ubuntu Zesty):
assignee: nobody → Dustin Kirkland  (kirkland)
status: New → In Progress
importance: Undecided → High
Changed in base-files (Ubuntu Zesty):
status: In Progress → Fix Committed
Changed in base-files (Ubuntu Artful):
status: In Progress → Fix Committed
summary: - 50-motd-news costs 5 seconds every login on firewalled systems
+ [SRU] 50-motd-news costs 5 seconds every login on firewalled systems
description: updated
Robie Basak (racb) wrote :

16:03 <rbasak> kirkland: ^ what happens on first login? That'll still be delayed, right? So the bug is being reduced in scope, but not being fixed entirely?

16:03 <rbasak> If the first login is before the timer has run for the first time, that is, which seems likely.

16:04 <kirkland> rbasak: true

16:04 <kirkland> rbasak: that's an easy fix, though, just one more line to add

16:05 <kirkland> rbasak: would you like me to fix that?

16:05 <kirkland> I think it would be a good idea, yeah

16:05 <rbasak> kirkland: yes please. Probably better for the SRU process to do it fewer times :)

description: updated
Changed in base-files (Ubuntu Zesty):
status: Fix Committed → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package base-files - 9.6ubuntu101

---------------
base-files (9.6ubuntu101) artful; urgency=medium

  * update-motd.d/50-news: Fix LP: #1691901
    - when the curl fails (perhaps due to a firewall issue), we should
      create an empty cache file, so that subsequent runs are not
      penalized; ensure that we *only* print when we're not forced

 -- Dustin Kirkland <email address hidden> Thu, 29 Jun 2017 10:35:06 -0500

Changed in base-files (Ubuntu Artful):
status: Fix Committed → Fix Released

Hello Scott, or anyone else affected,

Accepted base-files into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-files/9.6ubuntu13.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in base-files (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-zesty

The fix for this bug has been awaiting testing feedback in the -proposed repository for zesty for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Steve Langasek (vorlon) wrote :

I've run through the test case as described, and confirm that this takes care of the login delay problem.

tags: added: verification-done-zesty
removed: verification-needed verification-needed-zesty
tags: removed: removal-candidate

The verification of the Stable Release Update for base-files has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package base-files - 9.6ubuntu13.1

---------------
base-files (9.6ubuntu13.1) zesty-proposed; urgency=medium

  * update-motd.d/50-news: Fix LP: #1691901
    - when the curl fails (perhaps due to a firewall issue), we should
      create an empty cache file, so that subsequent runs are not
      penalized; ensure that we *only* print when we're not forced

 -- Dustin Kirkland <email address hidden> Thu, 29 Jun 2017 09:22:56 -0500

Changed in base-files (Ubuntu Zesty):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers