I noticed quite a long time to login to some of my systems via ssh (or scp).
Investigating lead me to find out that the '50-motd-news' file
(/etc/update-motd.d/50-motd-news) was the primary cost.
In line 108 [1] it does something like:
curl --connect-timeout "5" --max-time "5" -A "..." -o - https://motd.ubuntu.com
The systems I'm seeing this on are in a lab and do not have access to https://motd.ubuntu.com.
The way the lab is configured, they just end up timing out. So every scp or ssh connection
or other path to trigger update-motd will cost 5 seconds.
======= SRU =======
[ IMPACT ]
This bug affects Ubuntu 17.04 systems which cannot reach the internet (more specifically, https://motd.ubuntu.com). There is a bug in the implementation of /etc/update-motd.d/50-news whereby the failure of the curl fetch of the motd news causes a 5-second delay. This should not be the case. By design, the motd is fetched in the background, by a systemd timer, and refreshed every ~12 hours, also in the background. On login, the script should just cat the cache file. If the cache file isn't found, then it will try to fetch a new motd. In the firewalled system case, the background fetch failed, AND didn't create an empty file. The fix as attached simply ensures that an empty file is created if the curl fetch fails. This eliminates the login delay and solves the bug. Every 12 hours, the curl will be retried, quietly in the background, so if the curl failure happened for temporary reasons, it'll recover, gracefully, eventually.
[ TEST CASE ]
You can either test this on a firewalled system. Or, if you can hack an entry in your local /etc/hosts for motd.ubuntu.com for an invalid IP address. Without the fix, you'll experience a 5 second delay on login. With the fix, you'll login immediately.
Failure case:
$ lxc launch ubuntu:17.04 LP1691901
$ lxc exec LP1691901 bash
# ssh-import-id kirkland
# echo 192.168.0.0 motd.ubuntu.com >> /etc/hosts
# rm -f /var/cache/motd-news
# exit
$ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true
real 0m5.333s
user 0m0.016s
sys 0m0.000s
Apply the fix.
$ time ssh root@$(lxc list | grep LP1691901 | awk '{print $6}') true
real 0m0.316s
user 0m0.008s
sys 0m0.008s
[ REGRESSION ]
This is a simple, safe fix with minimal regression potential:
diff -Nru base-files-9.6ubuntu13/update-motd.d/50-motd-news base-files-9.6ubuntu13.2/update-motd.d/50-motd-news
--- base-files-9.6ubuntu13/update-motd.d/50-motd-news 2017-02-15 13:54:04.000000000 -0600
+++ base-files-9.6ubuntu13.2/update-motd.d/50-motd-news 2017-06-29 10:43:05.000000000 -0500
@@ -51,9 +51,13 @@
# If we're not forcing an update, and we have a cached motd-news file,
# then just print it and exit as quickly as possible, for login performance.
# Note that systemd should keep this cache file up to date, asynchronously
-if [ "$FORCED" != "1" ] && [ -r $CACHE ]; then
- echo
- safe_print $CACHE
+if [ "$FORCED" != "1" ]; then
+ if [ -r $CACHE ]; then
+ echo
+ safe_print $CACHE
+ else
+ : > $CACHE
+ fi
exit 0
fi
@@ -111,7 +115,9 @@
safe_print "$NEWS"
# Try to update the cache
safe_print "$NEWS" 2>/dev/null >$CACHE || true
- fi
+ else
+ : > "$CACHE"
+ fi
done
rm -f "$NEWS" "$NEWS.err"
exit 0
--
[1] https://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files/tree/update-motd.d/50-motd-news?h=applied/ubuntu/zesty#n108
ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: base-files 9.6ubuntu13
ProcVersionSignature: User Name 4.10.0-21.23-generic 4.10.11
Uname: Linux 4.10.0-21-generic x86_64
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
Date: Fri May 19 01:11:30 2017
Ec2AMI: ami-0000004f
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: base-files
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.update-motd.d.50-motd-news: 2017-05-19T01:10:25.572110
I've adjusted the bug description to match my understanding of the actual problem.
AIUI this does not happen on a disconnected system; if motd.ubuntu.com cannot resolve or returns ECONNREFUSED or EHOSTUNREACH, there should be no 5-second delay. The error only happens on a system which has routes, but has a firewall that drops packets instead of responding to the request. (Or when motd.u.c is actually down.)
So, what you appear to be describing here is a deliberate design tradeoff. Unless you are recommending a different timeout (if so, what value?) or asking for this to be disabled altogether (in which case I will refer you to Dustin), I think the only solution here is for your lab to have suitable firewall+proxy settings.