apt-helper / update-notifier / ttf-mscorefonts-installer download failure (regression)

[Adrien Beau]

* OS release: Ubuntu 16.04.1 LTS
* Package version: 3.168.3

For the past few days (maybe even weeks for other users), update-notifier has been very repeatedly failing to handle the downloads for ttf-mscorefonts-installer. Previously it was working fine on the same system.

This seems to affect a lot of desktop users, but the issue is usually misdiagnosed on forums because it is attributed to ttf-mscorefonts-installer and not to update-notifier-common.

The issue is made particularly irritating because the update-notifier cron job tries and fails and notifies users daily of this failure.

The issue can be reproduced by having ttf-mscorefonts-installer installed, and running:

    sudo dpkg-reconfigure update-notifier-common

The error messages seem to be random but actually appear to be consistent for a given download source:

* freefr.dl.sourceforge.net: 404 Not Found
* kent.dl.sourceforge.net: Protocol "http" not supported or disabled in libcurl
* netassist.dl.sourceforge.net: The HTTP server sent an invalid Content-Range header Bad header data [IP: 62.205.134.42 80]
* netcologne.dl.sourceforge.net: The HTTP server sent an invalid Content-Range header
* netix.dl.sourceforge.net: 404 Not Found
* vorboss.dl.sourceforge.net: 404 Not Found

In all above cases, the reported failing URL can be downloaded just fine using wget.

Some other mirrors appear to work (unfortunately, mirror names are only printed in case of errors). However, since ttf-mscorefonts-installer triggers 11 downloads in a row, chances of randomly getting 11 working mirrors are very low.

I am tagging this as regression-update because the issue started happening when upgrading from 3.168.2 to 3.168.3 last Friday. It is of course entirely possible the root cause is in another package, but this would also likely be a regression-update, since I am very careful about system-wide changes on this machine.

The previous update from 3.168.1 to 3.168.2 on November 4th went just fine. ttf-mscorefonts-installer has been installed (just fine) on September 26th, and not upgraded since then.

[Adrien Beau (adrienbeau)]

Digging into update-notifier-common, it appears the actual failure can be reduced to the following command-line (for 1 of the 11 downloads):

/usr/lib/apt/apt-helper \
  download-file \
  http://downloads.sourceforge.net/corefonts/andale32.exe \
  /tmp/andale32.exe \
  SHA256:0524fe42951adc3a7eb870e32f0920313c71f170c859b5f770d82b4ee111e970

Output for a mirror that works:

Get:1 http://downloads.sourceforge.net/corefonts/andale32.exe [198 kB]
Fetched 198 kB in 2s (94,3 kB/s)

Output for a mirror that fails:

Err:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  The HTTP server sent an invalid Content-Range header
E: Failed to fetch https://netcologne.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe The HTTP server sent an invalid Content-Range header
E: Download Failed

Interestingly, running apt-helper with the redirected URL, after taking care to %-encode the space in the URL, works!

/usr/lib/apt/apt-helper \
  download-file \
  https://netcologne.dl.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe \
  /tmp/andale32.exe \
  SHA256:0524fe42951adc3a7eb870e32f0920313c71f170c859b5f770d82b4ee111e970

But it gets weirder!

Running the very same command (downloading again from netcologne) results in the "invalid Content-Range" failure. Removing the target file, the download works again.

Running the command for another mirror (netix instead of netcologne) first works (same as netcologne). But then, running the very same command results in a "Protocol "http" not supported or disabled in libcurl" error! Removing the target file, the download works again.

To summarize my findings:

* Running apt-helper on http://downloads.sourceforge.net/corefonts/andale32.exe most often fails (it seems to handle the redirect to the mirror, but maybe fails to encode the space in the redirect URL)
* Running apt-helper on the mirror URL (with %20 instead of space in the URL) works if and only if the target file on the local system does not exist
* Running apt-helper on the mirror URL when the target file already exists results in various mysterious errors

All in all, it looks to me like apt-helper is a bit broken.

apt-helper is part of the apt package. The version currently on my system is 1.2.15ubuntu0.2. It was updated from 1.2.15 to 1.2.15ubuntu0.2 on December 14th.

[Adrien Beau (adrienbeau)]

The following recent bugs are quite likely related to this one:

* https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1646737
* https://bugs.launchpad.net/ubuntu/+source/update-notifier/+bug/1641671
* https://bugs.launchpad.net/ubuntu/+source/msttcorefonts/+bug/1607535 (huge, hot one)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Elias Kouskoumvekakis (eliaskousk)]

That's great news, thank you! At last we have the root cause.

I can confirm your findings on my Kubuntu 16.04 LTS.

[Adrien Beau (adrienbeau)]

Investigating further, it is quite possible the problem appeared in APT 1.2.15. Timeline of install and updates on my machine:

* 2016-07-19: U apt:amd64 1.2.10ubuntu1 -> 1.2.12~ubuntu16.04.1 # OS install
* 2016-09-26: I update-notifier-common:amd64 3.168.1 (automatic)
* 2016-09-26: I ttf-mscorefonts-installer:amd64 3.4+nmu1ubuntu2 # no problem
* 2016-11-04: U update-notifier-common:amd64 3.168.1 -> 3.168.2 # no problem
* 2016-11-14: U apt:amd64 1.2.12~ubuntu16.04.1 -> 1.2.15
* 2016-12-14: U apt:amd64 1.2.15 -> 1.2.15ubuntu0.2
* 2017-01-06: U update-notifier-common:amd64 3.168.2 -> 3.168.3 # PROBLEM

Between the time it ran with no problem on 2016-11-04 and the time the problem appeared, APT received two updates. According to the ChangeLog, the second one was a rather focused security fix. It is more likely that the first one is at fault, this it contains many minor changes.

[Adrien Beau (adrienbeau)]

Investigations (comments #1 and #5) indicate that the apt package is the most likely culprit. Quite possibly the update to 1.2.15 on Xenial caused the regression.

[Julian Andres Klode (juliank)]

There is no regression. Period. It's a myth.

Sourceforge changed to include a space in some redirects, braking the whole damn thing because we decode the redirect and https does not reencode it.

Marking as duplicate of 1651923.

[Adrien Beau (adrienbeau)]

I strongly disagree!

The space was put in the URL by the original corefonts SourceForge project way back in 2002, and has been there since then!

The space was there when apt 1.2.12~ubuntu16.04.1 successfully handled the redirects on 2016-09-26 and on 2016-11-04.

Now apt 1.2.15ubuntu0.2 fails the handle the space, and that is quite simply called a regression.

Also, the failure that I report is more complex than mishandling space encoding, see the bottom of comment #1 for the weirder part (various HTTP failures depending on.. the presence of the target file on disk).

[Julian Andres Klode (juliank)]

That does not really matter if it was there before or not. The problem is that there is now a redirect. That apparently was not there before.

In any case, this might take months to resolve for xenial - I have continously pushed updates for the past 2 months, but none of them has been accepted yet (one has finally been approved for -proposed, but that was two weeks ago already).

The other thing about the content range is a separate issues. The servers respond with
 "Content-Range: bytes */198384" on the 302 redirect. Which makes no real sense, but oh well.

[Adrien Beau (adrienbeau)]

I am certain the redirect was there before.

The redirector and the mirrors are listed in the Debian package ttf-mscorefonts-installer version 3.6, and were added back in 2013 or 2014, with a comment saying "we just use SF.net's redirection, which will work in most cases. The others serve as fallbacks to retry."

I also have always seen SourceForge using these mirrors (it was already redirecting to them in 2002...), though it is possible the technical details of the redirect have changed over time.

Anyway, I understand and agree that solving the URL decoding/re-encoding issue is the most important step, and thank you for working on it. It is too bad that it might take months to land on Xenial, considering how much heat #1607535 got in the past few days (and so much duplicates!).

There is still something in this bug report (in addition to the Content-Range weirdness) that is not a duplicate. This works the first time it is run:

/usr/lib/apt/apt-helper \
  download-file \
  https://netix.dl.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe \
  andale32.exe \
  SHA256:0524fe42951adc3a7eb870e32f0920313c71f170c859b5f770d82b4ee111e970

... and fails every time it is re-run, until andale32.exe is removed.

Should I file a separate issue?

[Julian Andres Klode (juliank)]

That is a problem on sourceforge:

E: Failed to fetch https://netix.dl.sourceforge.net/project/corefonts/the%20fonts/final/andale32.exe Redirection from https to 'http://downloads.sourceforge.net/mirrorproblem?failedmirror=netix.dl.sourceforge.net' is forbidden

We send it a valid https request, indicating that we already have 198384 bytes (the complete file)

> GET /project/corefonts/the%20fonts/final/andale32.exe HTTP/1.1
Host: netix.dl.sourceforge.net
User-Agent: Debian APT-CURL/1.0 (1.4~beta2)
Accept: */*
Cache-Control: max-age=0
Range: bytes=198384-
If-Range: Thu, 15 Aug 2002 14:33:49 GMT

It answers it has found the file at another location, but redirects to an unencrypted mirror problem URL:

< HTTP/1.1 302 Found
< Date: Wed, 11 Jan 2017 12:30:56 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips
< Location: http://downloads.sourceforge.net/mirrorproblem?failedmirror=netix.dl.sourceforge.net
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=iso-8859-1
<

There was another issue with "invalid content-range" - I'm working on that. But that one can easily be resolved on the redirector as well. It makes no sense for the server to respond with "Content-Range: */<file size>" (basically meaning: Oh, I can't satisfy that range) on a redirect.

[Jonathan Kamens (jik)]

The bug of which this one was previously marked a duplicate (I unmarked it) was fixed months ago, and yet I'm still getting this email from cron.daily every single day:

>/etc/cron.daily/update-notifier-common:
>ttf-mscorefonts-installer: processing...

The problem at this point seems to have nothing whatsoever to do with spaces in URLs or anything like that. Rather, the problem is that when the package ttf-mscorefonts-installer is installed non-interactively, the license is not accepted, and the downloader for the package silently fails to download any fonts if the license hasn't been accepted.

Note this stanza at the end of /usr/share/package-data-downloads/ttf-mscorefonts-installer:

>Script: /usr/lib/msttcorefonts/update-ms-fonts
>Should-Download: msttcorefonts/accepted-mscorefonts-eula

On my system, msttcorefonts/accepted-mscorefonts-eula is false, because the package was installed non-interactively.

The code which silently refuses to download fonts if that setting is false is in /usr/lib/update-notifier/package-data-downloader:

> if 'Should-Download' in para:
> db = debconf.DebconfCommunicator('update-notifier')
> try:
> should = db.get(para['Should-Download'])
> if should == "false":
> # Do nothing with this file.
> break
> except:
> pass
> finally:
> db.shutdown()

Furthermore, maybe I'm missing something, but I can't find any supported way of accepting this license once the package has been installed, because the code that prompts the user to accept the license is in /var/lib/dpkg/info/ttf-mscorefonts-installer.preinst, and as far as I can tell, dpkg-reconfigure doesn't rerun preinst files.

[LinuxRussia (kobylyanets)]

:~$ sudo apt install ttf-mscorefonts-installer --reinstall
Чтение списков пакетов… Готово
Построение дерева зависимостей
Чтение информации о состоянии… Готово
Обновлено 0 пакетов, установлено 0 новых пакетов, переустановлено 1 пакетов, для удаления отмечено 0 пакетов, и 0 пакетов не обновлено.
Необходимо скачать 0 B/27,6 kB архивов.
После данной операции, объём занятого дискового пространства возрастёт на 0 B.
Предварительная настройка пакетов ...
(Чтение базы данных … на данный момент установлено 185726 файлов и каталогов.)
Подготовка к распаковке …/ttf-mscorefonts-installer_3.6ubuntu2_all.deb …
mscorefonts-eula license has already been accepted
Распаковывается ttf-mscorefonts-installer (3.6ubuntu2) на замену (3.6ubuntu2) …
Обрабатываются триггеры для update-notifier-common (3.186) …
ttf-mscorefonts-installer: processing...
ttf-mscorefonts-installer: downloading http://downloads.sourceforge.net/corefonts/andale32.exe
Ошк:1 http://downloads.sourceforge.net/corefonts/andale32.exe
  Redirection from https to 'http://downloads.sourceforge.net/mirrorproblem?failedmirror=vorboss.dl.sourceforge.net' is forbidden
E: Не удалось получить https://vorboss.dl.sourceforge.net/project/corefonts/the fonts/final/andale32.exe Redirection from https to 'http://downloads.sourceforge.net/mirrorproblem?failedmirror=vorboss.dl.sourceforge.net' is forbidden
E: Ошибка при скачивании
Настраивается пакет ttf-mscorefonts-installer (3.6ubuntu2) …
Обрабатываются триггеры для fontconfig (2.11.94-0ubuntu2) …

[Julian Andres Klode (juliank)]

This problem has been fixed a while ago, seems we forgot to close it. Obviously Sourceforge is having issues again, causing problems for LinuxRussia (kobylyanets).

[Julian Andres Klode (juliank)]

@Jonathan Kamens (jik) The problem with msttcorefonts/accepted-mscorefonts-eula is a different one, you could open a bug for msttcorefonts for that. Silently not doing it seems to be the right thing, though. If you want to install it non-interactive, you could preseed the debconf value somehow.

[Julian Andres Klode (juliank)]

Let's mark this as a duplicate again, so we actually have stuff correct again.

[Tim Ritberg (xpert-reactos)]

Still a bug in 17.10

[Boris Rybalkin (ribalkin)]

Was able to fix sf mirrors issue by replacing sf mirror with some github copy:

sudo sed -i 's#http://downloads.sourceforge.net/corefonts#https://github.com/pushcx/corefonts/raw/master#g' /usr/share/package-data-downloads/ttf-mscorefonts-installer

Then run:
sudo /usr/lib/update-notifier/package-data-downloader

All downloaded and good.