motd.ubuntu.com currently shows media item (HBO's Silicon Valley using Ubuntu)

[Zachary Fouts]

In Ubuntu 17.04 or newer, there is a script at /etc/update-motd.d/50-motd-news that reads https://motd.ubuntu.com/ and displays that text with the rest of the MOTD.

Currently, https://motd.ubuntu.com shows a news item about HBO's Silicon Valley which has a reference to Ubuntu.
Instead, https://motd.ubuntu.com should show relevant items to those that
use Ubuntu Server (relevant security issues, etc), instead of items for desktop users.

=============================
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-21-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

 * How HBO's Silicon Valley built "Not Hotdog" with mobile TensorFlow,
   Keras & React Native on Ubuntu
   - https://ubu.one/HBOubu
==============================

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: base-files 9.6ubuntu13
ProcVersionSignature: Ubuntu 4.10.0-24.28-generic 4.10.15
Uname: Linux 4.10.0-24-generic x86_64
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
Date: Wed Jun 28 12:31:24 2017
InstallationDate: Installed on 2017-05-02 (56 days ago)
InstallationMedia: Xubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: base-files
UpgradeStatus: No upgrade log present (probably fresh install)

[Andrew Starr-Bochicchio (andrewsomething)]

I have to say, I was very disappointed to see this when I logged onto one of my servers today. I can understand the desire to be able to communicate directly to users and present timely, relevant information, but linking out to content marketing in what seems to be one of its first uses is self-sabotage. This type of behavior will lead to it being disabled and the "important security messages" to not be seen.

Note, you can disable this behavior in /etc/default/motd-news

[Simos Xenitellis  (simosx)]

Which RSS feed is shown here?

[Mikko Tanner (shapemaker)]

I strongly agree with @andrewsomething here. Advertising has absolutely no place in motd.

[Jonathan Carter (jonathan)]

@Mikko It should really have no place at all in something that you'd like to call a free operating system. Unfortunately Canonical strongly disagrees with that position.

[Simos Xenitellis  (simosx)]

The script is /etc/update-motd.d/50-motd-news
which reads https://motd.ubuntu.com and displays the text from that page.

Obviously there has been an unfortunate mix-up and it shows that HBO-related item.

Is it really necessary to conflate this into some conspiracy to display ads in the Ubuntu Server motd?

[Simos Xenitellis  (simosx)]

Also, this is not an issue with base-files, but rather with motd.ubuntu.com.

The proper place to file this, would be somewhere at https://github.com/canonical-websites

[Sander Steffann (sander-steffann)]

Well, it's acombined issue:

An ubuntu package shows content from motd.ubuntu.com. If we have control over what motd.ubuntu.com contains then we should fix that. If we don't have control over the quality of what motd.ubuntu.com contains then we shouldn't use it.

User's won't know where the information comes from, only that Ubuntu shows it. Therefore we should only show information from sources we can rely on to show useful information to users. Either the source should be fixed, or it shouldn't be used...

[M. Lawson (mlawson1986)]

While I agree this shouldn't be seen as some sort of conspiracy, there are a few things I want to comment on:

1) This shouldn't happen in the first place. ubuntu.com is not ad supported, and neither should the OS. Canonical makes its money through support services and should not need the ad revenue. If for some reason they do, a formal statement needs to be released

2) This is kind of sneaky to put in to the motd

3) Regardless of where it's coming from, ubuntu.com is the domain. That's where the onus lay.

[Joonas Saarinen (jza)]

Even if it isn't seen as an advertisement, it's a bit questionable and unprofessional to put cool facts about a TV show in a space reserved for technical announcements.

[Dustin Kirkland  (kirkland)]

Howdy! I'm the author of the motd-news code, as well as the story in the subject of this bug.

Note that motd.ubuntu.com has been updated to say:

" * Ubuntu 16.10 will reach end of life on Wednesday, July 20th"

It may be useful to explain a bit of the design of motd-news. Note that much of this was copied over from my comment on the Hacker News thread: https://news.ycombinator.com/edit?id=14663947

By design:

- Asynchronously, about 60 seconds after boot, a systemd timer fires which runs "/etc/update-motd.d/50-motd-news --force"

- It sources 3 admin-editable config variables in /etc/default/motd-news. The defaults are: ENABLED=1, URLS="https://motd.ubuntu.com", WAIT="5"

- The admin can disable it entirely (ENABLED=0), change or add other MOTD news sources (your corporate IT team could run its own), and change the wait time in seconds

- If it's enabled, that systemd timer job will loop over each of the URLS (note, that it's important that these should be https with valid SSL certificates), trim them to 80 characters per line, and a maximum of 10 lines, and concatenate them to a cache file in /var/cache/motd-news

- Every ~12 hours thereafter (with a little bit of random timer fuzzing), this systemd timer job will re-run and update the /var/cache/motd-news

- Upon login, the contents of /var/cache/motd-news is just printed to screen. There was a bug (fixed now, working its way through the Ubuntu update process), in that an empty cache file was not getting, if the fetch failed, so the fetch was retried every time (and failed every time, with the $WAIT delay of 5 seconds). Obviously, it was a simple fix.

As I am generally responsible for this feature in Ubuntu (as the Ubuntu Product Manager for the design, and the Ubuntu Core Developer for the implementation) I'll also respond to the discussion about this feature being included in Ubuntu at all...

- Back in 2009, Ubuntu was the first distribution to add the concept of a "dynamic MOTD", by introducing the /etc/update-motd.d/* structure of scripts. I originally implemented it for Landscape (a commercial package management system by Canonical), but it has since also been adopted by Debian, Amazon Linux, and others.

- It's a flexible framework that enables distro packages or administrators to add executable scripts in /etc/update-motd.d/* to generate informative, interesting messages displayed at login.

- This was quite interesting, in that for almost 40 years of Linux/UNIX, the "Message of the Day" was anything but that... It was a message that was created at one point in time, when the distro released, and that's about it. And we managed to change that.

- If you look in your Ubuntu's /etc/update-motd.d/, you'll see a set of scripts. One prints the generic "welcome" banner. The next one prints 3 links, as to where to find help. You'll also find one that counts and displays the number of package updates available for the local system. Another tells you if a reboot is required.

- All of these are locally generated, based on local information available on the system.

- In Ubuntu 17.04, we added /etc/update-motd.d/50-motd-news. This is the script who's design is described above...

[Zachary Fouts (zfouts)]

Dustin -- thank you for addressing the issue.

[Timothy R. Chavez (timrchavez)]

Just my opinion here..

```
* Sometimes, there's critical information that either we, the distro, or perhaps even the local IT administrator of a fleet of systems needs to communicate to users at login. Think, "ShellShock", "Heartbleed", "StackClash". This seems to happen every few months, in the current state of the world.

* Other times, we need to communicate something about the distribution. EoL messages, new feature availability, etc.
```

^^^ Great use of motd-news

```
And sometimes, it's just a matter of presenting a fun fact.
```

^^^ Not a great use of motd-news

I wouldn't want to have to opt out of receiving critical information or EOL messages just to avoid receiving fun facts. Can we maybe split off the fun fact stuff into it's own motd-funfact script and have it disabled by default on the server image?

[Timothy R. Chavez (timrchavez)]

I don't know if this should be a separate issue, but while investigating this issue and learning about motd-news, I stumbled across this in the code https://gist.github.com/anonymous/fdc1cab8cb193ca19aa4c663c1ebd1f5#file-gistfile1-txt-L245 -- While I applaud the creativity, I don't think it's right for Canonical to be collecting server information into the USER_AGENT and curl'ing it back to motd.ubuntu.com. First, it's completely non-obvious. No one would be the wiser that Canonical is collecting this information if they weren't actually looking at the code. Second, you have to opt-out of it. Third, by opting out of it, you're also forced to opt out of critical information about vulnerabilities, distro EOL messages, etc. In this way, we're sort of having to "buy" motd-news with information about our servers. That should be spelled out somewhere, at the very least.

[Timothy R. Chavez (timrchavez)]

I should walk back some of what I've said. I don't want to be inflammatory. I implied Canonical was collecting this information when they may only in fact be using it to decide which messages to send back to the client. It's entirely possible that the information is being "thrown away" after the http response is sent. If that's the case, it would be great to state it somewhere. Ideally it would be nice to make all those decisions locally, but I recognize the trade offs in terms of payload size and complexity. My apologies

[Horatio Trobinson (horatio-trobinson)]

1) This is ideological content and it's not being used to disseminate technical or factual knowledge that is useful to the user.

I would be less inclined to be critical if the non-relevant information delivered was content that helped to amplify the work of a world-renowned charity (say, MSF for example) butthe relevant news I've seen are just part of a certain Silicon Valley-dominated ideology, and as such it presents the potential of being inadvertently used as a propaganda vehicle to advance particular interests of a specific group of people.

2) It doean't seem that enough precautions have been taken to avoid makibg motd a central point of failure. It doesn't seem to take into accoubt that there's a potebtial for privacy and exploit problems generated for desktop users, without offeribg a straightforward opportunity to opt out. It's a practice that should at a minimum be part of an optional and customisable policy kit in enterprise environments.

3) For technical reasons and other reasons explained above, I respectfuly suggest that this practice should be completely reconsidered in further detail or discontinued altogether.

[Simos Xenitellis  (simosx)]

@Timothy:

There would not be much of a need to collect stats specifically from motd.ubuntu.com. The user-agent will just say 'curl' in all cases which is not that helpful.
Also, each distro installation checks every day for security updates, so why would anyone bother with motd?

There has been a sad Twitter storm in a tea cup over this, and it makes otherwise nice people to be negative about Ubuntu.

[Timothy R. Chavez (timrchavez)]

@Simos:

Not sure about the Twitter storm you're referring too. The http request being sent to motd.ubuntu.com is embedding information like system uptime in the user-agent header. That appears to be used to determine which information to send back in the http response. What's not clear, is if it's being collected or not. Also, while unattended-upgrades for -security may be a default, it may also be disabled, so I don't think you can assume that. That said, I do think it's a good point that if you're automatically receiving security upgrades, how much the motd news that covers such things actually matters. Is your recommendation to disable motd-news by default and make it opt-in?

[Dustin Kirkland  (kirkland)]

This comment is mainly for Tim and Andrew (of Digital Ocean), who seem offended by this behavior.

I just captured this snapshot in my Chrome browser, searching for "Docker Cloud". The top results of that search are not actually the organic docker.com results, but rather a Google Ad for Digital Ocean services.

Moreover, the HBO link wasn't even an advertisement! It was an interesting piece of news from this week, a blog post of an interesting developer who did something fun on top of Ubuntu. And we celebrated that achievement.

[Dustin Kirkland  (kirkland)]

Regarding the User Agent -- this is exactly equivalent to the USER AGENT string your browser (Chrome, Chromium, Firefox, Safari, curl, wget) sends to every website in the world. Curious? Install apache2 on localhost, tail -f /var/log/apache2/access.log, and hit http://localhost/ with a web browser. Here, I've just done it for you:

127.0.0.1 - - [29/Jun/2017:17:55:20 -0500] "GET / HTTP/1.1" 200 777 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36"

[Dustin Kirkland  (kirkland)]

Regarding the "sometimes the news is just a fun fact"...

Do you get similarly annoyed when the Google logo on the homepage is replaced with a Google Doodle?

https://www.google.com/doodles

[Zachary Fouts (zfouts)]

Re: UserAgent
My UserAgent in Chrome|FireFox|Curl|wget|etc do not tell the remote server what the uptime of my host is.

Re: Fun Facts
Fun facts are indeed fun, but this feature should be reserved for important information regarding EOL, Security Patches, etc.
If the administrator of ${system} wants a fun fact, they can install something else. Cow Say, Fortune, whatever to display that.

Not trying to stir anything up, it's a great feature but that feature should be used wisely so people do not disable it.

[Timothy R. Chavez (timrchavez)]

@Dustin

Are we looking at the same USER_AGENT string? I ran the block of code from `update-motd.d/50-motd-news` on my laptop that constructs the USER_AGENT variable that's sent to motd.ubuntu.com and it looks like this: "curl/7.47.0-1ubuntu2.2 Ubuntu/16.04.2/LTS GNU/Linux/4.7.0-040700-generic/x86_64 Intel(R)/Core(TM)/i7-7500U/CPU/@/2.70GHz uptime/1499911.84/3827524.62" not what's described at https://en.wikipedia.org/wiki/User_agent and not what you pasted above. Reading the comments in the code, it's clear that it's being used to craft the response, but what's not clear is if Canonical is collecting the server information... If they are, I think it's fair to make that apparent. That's all.

[M. Jensen. (newsworthy39)]

+1 Thanks, for briging this important security information to me. I'll remove the motd-file, to prevent tracking and the likes.

[Nicola Heald (notnownikki)]

@Dustin

> Moreover, the HBO link wasn't even an advertisement!

I think the thing that made me feel uneasy is that the motd read like an advertisement. And so did parts of the article, specifically saying that we should watch Silicon Valley. I appreciate that it was not meant that way though.

But maybe people are so sick of seeing clickbait advertorial content when they browse the internet that the message brought up some bad reactions. Perhaps people thought that, when the linked article told them that they should watch Silicon Valley, that it was promoting Silicon Valley. Which is a great show on HBO. If you haven't seen it, you should! (This is not a promotion of Silicon Valley.)

[Markus Ueberall (ueberall)]

Just my 2 cents:
In general, publishing important security/maintenance related notices by means of https://motd.ubuntu.com sounds like a good idea (especially if you don't use canonical-livepatch in case of critical security hazards), but I'd strongly suggest to allow for client side configuration/filtering by means of tags, e.g.,

 * [16.10][maintenance] Ubuntu 16.10 will reach end of life on […] July 20, 2017
 * [16.04][security][<4.4.0-83] Please consider updating your kernel ASAP

(The second example would require version based comparisons, but you get the idea.)
There's no need to disclose any client specific information, and it won't hurt to transfer a few bytes more.

[Thorsten Glaser (mirabilos)]

 “The default configuration is that this "motd-news" feature is enabled and that it will check https://motd.ubuntu.com for updates.” is called “a useful feature” by the author of the LWN article, and in this bugreport.

In the Debian world, we call this a phone-home privacy violation which is a security-relevant release-critical bug and an absolute MUST NOT.

[Marco van Zwetselaar (zwets)]

I actually spent time tracking down this 'bug' because on my upgrade 17.04 servers I kept seeing:

  * Ubuntu 16.10 will reach end of life on Thursday, July 20, 2017
     How to upgrade from 16.10 to 17.04:
     - https://ubu.one/upgY2Z

making me wonder if my upgrade was botched or something. I have little against the feature in general (though I would certainly make it opt-in rather than opt-out), but this motd is confusing.

[Nish Aravamudan (nacc)]

@zwets: agreed, I see that as well. I assume the server side of motd.ubuntu.com is not using the USER_AGENT lsb value correctly?

[Nish Aravamudan (nacc)]

@zwets, would you be willing to file a new bug? It's basically a clarity issue on the messsage itself. It isn't actually saying you are running 16.10 or 17.04 or anything else. It's just letting you know that 16.10 goes EOL soon. So update your machines. But not necessarily that this particular machine needs updating. It's confusing and I think deserves a bit of rewriting to be clearer (maybe using the lsb user_agent to change the exact text of the message).

[Nish Aravamudan (nacc)]

@zwets, to be clear, file the bug at: https://bugs.launchpad.net/~ubuntu-motd