jigdo: files missing on archive http://archive.ubuntu.com/ubuntu/ - cannot assemble, e.g. ubuntu-16.04.3-server-amd64.iso

[Michael Paoli]

Of the 1474 files specified in
http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo
needed to assemble ubuntu-16.04.3-server-amd64.iso
210 of those files are missing from
http://archive.ubuntu.com/ubuntu/

It would appear this type of issue has existed or come up
before, e.g.:
ubuntu 10.04.1 jigdo images on mirrors are missing files
https://bugs.launchpad.net/ubuntu/+source/jigdo/+bug/647762

[Michael Paoli (michael-paoli)]

some more details can be found here:
https://lists.ubuntu.com/archives/ubuntu-us-ca/2017-November/002826.html
https://lists.ubuntu.com/archives/ubuntu-us-ca/2017-November/002827.html

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1731762/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Michael Paoli (michael-paoli)]

The bug isn't in jigdo itself, but Ubuntu really wants/prefers bugs be filed against package, so, ... since it's at least related ...
Also, don't know about Ubuntu, but Debian has effectively pseudo-packages for filing bugs against things that are bugs, but aren't specific to package(s), e.g. such as an issue with data on a Debian repository or web/FTP server (e.g. missing, outdated, or incorrect information, typo, ...) - and being filed againt appropriate pseudo-package, it gets routed to relevant developer(s)/team(s).
Feel free to update the Package this bug is filed against if there's more suitable Package (or pseudo-Package, or suitable way to tag or whatever) so it gets to the relevant person(s)/team(s).

[Brian Murray (brian-murray)]

How could I reproduce this issue? I'm not familiar with jigdo.

[Brian Murray (brian-murray)]

Ah, I was able to recreate it.

--2017-11-13 10:20:57-- http://archive.ubuntu.com/ubuntu/pool/main/b/bind9/bind9_9.10.3.dfsg.P4-8ubuntu1.7_amd64.deb
Reusing existing connection to archive.ubuntu.com:80.
HTTP request sent, awaiting response... 404 Not Found
2017-11-13 10:20:57 ERROR 404: Not Found.

FINISHED --2017-11-13 10:20:57--
Total wall clock time: 1.0s
Downloaded: 2 files, 890K in 0.6s (1.41 MB/s)
Found 2 of the 212 files required by the template
Copied input files to temporary file `ubuntu-16.04.3-server-amd64.iso.tmp' - repeat command and supply more files to continue

-----------------------------------------------------------------
Aaargh - 210 files could not be downloaded. This should not
happen! Depending on the problem, it may help to retry downloading
the missing files.
Also, you could try changing to another Ubuntu server,
in case the one you used is out of sync.

However, if all the files downloaded without errors and you
still get this message, it means that the files changed on the
server, so the image cannot be generated.
As a last resort, you could try to complete the CD image download
by fetching the remaining data with rsync.

Press Return to retry downloading the missing files.
Press Ctrl-C to abort. (If you re-run jigdo-lite later, it will
resume from here, the downloaded data is not lost if you press
Ctrl-C now.)

[Brian Murray (brian-murray)]

My suspicion is that the jigdo file is a snapshot of the .deb files that appear on the iso and that because this is a point a release the jigdo file points to .deb files which used to exist in -updates or -security that have been superseded by newer versions. If that's correct a .jigdo file for point releases (16.04.3) is only useful for a very short period of time.

[Michael Paoli (michael-paoli)]

Also, similar from earlier:
https://lists.ubuntu.com/archives/ubuntu-us-ca/2014-December/002550.html
https://lists.ubuntu.com/archives/ubuntu-us-ca/2014-December/002551.html
https://lists.ubuntu.com/archives/ubuntu-us-ca/2014-December/002552.html
Also, a search, e.g. on Google, for, e.g.:
ubuntu jigdo "Aaargh -" "files could not be downloaded"
also shows a fair number of folks occasionally bump into this general type of
bug with Ubuntu, jigdo, and files missing from archive/repository.

[Michael Paoli (michael-paoli)]

recreate/show bug ... okay, I'm a bit late to the party on that (didn't see the
earlier comment about having recreated it 'till now), anyway, here's another
example recreating/demonstrating the bug (again, it's with the
Ubuntu archive / jigdo file, not jigdo/jigdo-lite itself):

This was captured with script(1), then processed through:
col -b
Then edited down a bit to remove lots of redundancy (left to show first and last of
the 210 consecutive failures), and I added some comments with lines starting with:
//
for some additional clarity. Essentially I was using older (ubuntu-16.04-server-amd64.iso)
images as base to attempt to create newer ubuntu-16.04.3-server-amd64.iso image via
jigdo-lite(1) and http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo

$ mkdir Ubuntu_bug_1731762
$ cd Ubuntu_bug_1731762
$ mkdir ubuntu-16.04-server-amd64
$ sudo mount -o loop,ro,nosuid,nodev /var/local/ISOs/older/ubuntu-16.04-server-amd64.iso ubuntu-16.04-server-amd64
$ jigdo-lite http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo

Jigsaw Download "lite"
Copyright (C) 2001-2005 | jigdo@
Richard Atterer | atterer.net
Loading settings from `/home/m/michael/.jigdo-lite'

Downloading .jigdo file
converted 'http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo' (ANSI_X3.4-1968) -> 'http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo' (UTF-8)
--2017-11-14 03:53:48-- http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo
Resolving releases.ubuntu.com (releases.ubuntu.com)... 2001:7b8:3:37::21:3, 91.189.88.166
Connecting to releases.ubuntu.com (releases.ubuntu.com)|2001:7b8:3:37::21:3|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 137794 (135K)
Saving to: 'ubuntu-16.04.3-server-amd64.jigdo'

ubuntu-16.04.3-serv 100%[=====================>] 134.56K -143KB/s in 0.9s

2017-11-14 03:53:49 (143 KB/s) - 'ubuntu-16.04.3-server-amd64.jigdo' saved [137794/137794]

-----------------------------------------------------------------
Images offered by `http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.jigdo':
  1: 'Ubuntu-Server 16.04.3 LTS "Xenial Xerus" - Release amd64' (ubuntu-16.04.3-server-amd64.iso)

Further information about `ubuntu-16.04.3-server-amd64.iso':
Generated on Tue, 01 Aug 2017 11:30:42 +0000

-----------------------------------------------------------------
If you already have a previous version of the CD you are
downloading, jigdo can re-use files on the old CD that are also
present in the new image, and you do not need to download them
again. Mount the old CD ROM and enter the path it is mounted under
(e.g. `/mnt/cdrom').
Alternatively, just press enter if you want to start downloading
the remaining files.
Files to scan: ubuntu-16.04-server-amd64

Downloading .template file
converted 'http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.template' (ANSI_X3.4-1968) -> 'http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.template' (UTF-8)
--2017-11-14 03:54:14-- http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.template
Resolving releases.ubuntu.com (releases.ubuntu.com)... 2001:67c:1360:8001::26, 91.189.88.166
Con...

[Michael Paoli (michael-paoli)]

I'd certainly "argue" :-) / advocate, that the .jigdo files and
archive(s)/repository(/ies) of Ubuntu ought be such that
jigdo-lite(1) (and/or other Jigdo clients) can successfully create the
relevant target .iso files. Even though there may have been
updates (patches / security fixes) since ubuntu-16.04.3-server-amd64.iso,
ubuntu-16.04.3-server-amd64.iso is still (at least as I'm writing this)
the most current ISO for that flavor/variant (Ubuntu-Server LTS amd64),
and ought not only be downloadable (as it is), but ought be able to
reassemble it via Jigdo (saving both clients, networks, and servers,
much bandwidth and time compared to entirely downloading the full
ISO file, which is in substantial part redundant with the earlier
ubuntu-16.04-server-amd64.iso file).
E.g. after otherwise managing to assemble/download, and then "seed" it
to BitTorrent - still quite popular (folks have already downloaded a
fair chunk of it from me in relatively little time) - so there's
certainly no lack of interest in obtaining the iso ... that's also
typically what folks would boot from if that's what they want to
install.
 ------------------------------------------------------------------------------
| Filename Size Download Upload |
...
| ubuntu-16.04.3-server-amd64.iso 825.0 M 0 B/s 0 B/s |
| ^--- download succeeded! 0.0 M 234.8 M |
...
 ------------------------------------------------------------------------------

As noted much earlier on:
https://lists.ubuntu.com/archives/ubuntu-us-ca/2014-December/002551.html
Ubuntu may still have the individual files downloadable elsewhere,
... that's a start ... but things need to properly line up between
what's in the .jigdo files, and where and how things are downloaded.

E.g. with Debian :-) one can still use jigdo to assemble iso images
back to or nearly to 3.0_r0 (Woody), which was released 2002-07-19,
by using the relevant .jigdo file(s) from:
http://cdimage.debian.org/mirror/cdimage/archive/

Ubuntu ought maintain its archives and .jigdo files so the .iso files
can be assembled with Jigdo clients (e.g. jigdo-lite). Failure to do so
just causes lots of repeat problems for various folks, and much losses
in efficiency (of downloads, bandwidth, people's time, etc.). This
really needs to be fixed systemicly and in the process/procedures, so
they're properly maintained and validated. Shouldn't keep having this
problem over and over and over again.

[Michael Paoli (michael-paoli)]

Here's an UGLY 8-O ... but effective :-/ workaround:

After the original attempt(s), which still gets many of the files,
but also fails to get many of the files (210 in our slightly earlier
example - presuming that's still the case or similarly/nearly so),
proceed as follows:

First of all, those attempts note above should be made, as there are
some files that can be downloaded from the regular repositories/mirrors,
that aren't available with the workaround - so use the workaround just
to complete the process of constructing the image.

From that earlier attempt, we should already have the .jigdo and
.template files - if we don't for any reason, then download them.

Once we have those files, we'll create a modified version of the
.jigdo file, most notably we'll take advantage of the earlier
information from:
https://lists.ubuntu.com/archives/ubuntu-us-ca/2014-December/002551.html
regarding:
https://launchpad.net/ubuntu/+archive/primary/+files/
... except jigo-lite doesn't want https URLs, but will work
fine with http URLs - which is fine in this case, as URLs starting with:
http://launchpad.net/ubuntu/+archive/primary/+files/
redirect where we need them to anyway (and even URLs starting with
https://launchpad.net/ubuntu/+archive/primary/+files/
do redirects).

So ... how do we modify the file specifically?
We change the pathname components, and the Server URL(s) ...
at least for the file types (extensions) that can be obtained via
http://launchpad.net/ubuntu/+archive/primary/+files/
based URLs ... which fortunately covers all the file types we
need (at least at the present time, anyway).

So, how to do the modification? Best done by program/script:
(also attached):
$ expand -t 2 < workaround
#!/bin/sh

# Don't misinterpret our sed(1) character classes/ranges:
LC_ALL=C; export LC_ALL

sed -e '
  : Top

  /^\[Parts\]$/b Parts
  /^\[Servers\]$/b Servers
  n
  b Top

  : Parts
  s/^\([-0-9A-Z_a-z]\{22\}=Debian:\).*\/\([^\/]*.\.u\{0,1\}deb\)$/\1\2/
  /^\[/{
    /^\[Parts\]$/!b Top
  }
  n
  b Parts

  : Servers
  s/^Debian=.*$/Debian=http:\/\/launchpad.net\/ubuntu\/%2barchive\/primary\/%2bfiles\//
  /^\[/{
    /^\[Servers\]$/!b Top
  }
  n
  b Servers
'

rename our original .jigdo file, note also that .jigdo files may be
optionally gzip compressed - if so save it in uncompressed form - whether
or not it's compressed can be checked with the file(1) command:
$ file ubuntu-16.04.3-server-amd64.jigdo
ubuntu-16.04.3-server-amd64.jigdo: ASCII text
$ mv ubuntu-16.04.3-server-amd64.jigdo ubuntu-16.04.3-server-amd64.jigdo.original
if we instead got something like:
ubuntu-16.04.3-server-amd64.jigdo.gz: gzip compressed data ...
We would instead do:
$ gzip -d < ubuntu-16.04.3-server-amd64.jigdo > ubuntu-16.04.3-server-amd64.jigdo.original
$

Now apply our workaround to create new modified .jigdo file:
$ ./workaround < ubuntu-16.04.3-server-amd64.jigdo.original > ubuntu-16.04.3-server-amd64.jigdo

Now use our new .jigdo file to get the missing file and complete
building the image:
$ jigdo-lite ./ubuntu-16.04.3-server-amd64.jigdo
...
When prompted for mirror, be sure to use:
https://launchpad.net/ubuntu/+archive/primary/+files/
e.g.:
Debian mirror [http://arch...