wget-udeb should install to /usr/bin/wget instead of /usr/bin/wget.gnu

[Mark Russell]

SRU justification:
This is a prerequisite for bug 833994, which is a customer requirement for Ubuntu 12.04.5. It's probably simplest to test them all together.

Original report:

In the Ubuntu raring (13.04) version of wget, there is a wget-udeb which installs its binary executable to /usr/bin/wget.gnu.

This is presumably done in order to not break any setups that depend on busybox's wget implementation.

However, since the primary reason wget-udeb exists in Ubuntu (wget-udeb is not built in Debian afaik) is because of the lack of SSL support in d-i and busybox-wget, it seems logical (to me) that it should overwrite the busybox wget symlink. You're choosing to opt-in to GNU wget, so you're already rebuilding d-i/debian-cd and therefore know you're somewhat on your own.

Unless there is a common use case I'm not considering where you want SSL support for something else, but somehow depend on the busybox implementation of wget for the debootstrap portion of the install.

What I expect to happen:
1) modify d-i source to include wget-udeb
2) rebuild d-i and point my sources to HTTPS repositories
3) install Ubuntu without fear of the traffic being snooped in transit

What happens instead:
1) modify d-i source to include wget-udeb
2) rebuild d-i and point sources to HTTPS repositories
3) install fails because d-i calls /usr/bin/wget which points to busybox (which has no SSL support)

Thanks for your time!

Please note: this suggestion is not intended to securely authenticate the repository; that's absolutely another issue. This is simply to address potential snooping of traffic in transit.

[Mark Russell (marrusl)]

debdiff for raring

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "wget.raring.1172101.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Mark Russell (marrusl)]

I believe it's debootstrap that relies on wget for transport in the installer. Since debootstrap usually uses the "real" wget outside of d-i, this appears to be a safe change to me. But I may be overlooking something about the d-i environment that makes it different.

[Mark Russell (marrusl)]

I probably picked a bad version number for the patch as I'd like to see this SRU'd to precise as well. I'll resubmit if the basic idea is accepted.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[lezbak (lezgin-bakircioglu)]

Related:
"debian-installer does not support https when using with preseed files"
https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/833994

[Colin Watson (cjwatson)]

While this is *probably* OK - I think the different name was overcautious - before accepting this somebody needs to build an image with wget.gnu moved to wget and test that a basic d-i installation still works. It's possible that there are interface differences sufficient to kill the installer.

[Mark Russell (marrusl)]

Perhaps it wasn't overcautious after all. Surprisingly, I ran into problems attempting to install from a mini.iso built with a modified wget-udeb. At first I thought it may have a been a quirk of saucy. I thought maybe daily quality didn't extend to the mini.iso... I hoped! But then I replicated the issue on precise.

Here is the process I used:

- download wget source
- modify line in debian/rules to make udeb binary /usr/bin/wget (not wget.gnu)
- build binaries
- download d-i source
- put new wget-udeb into build/localudebs
- modified build/pkg-list/base to include wget-udeb (I'm told this was needless)
- build binaries
- install KVM machine using build/dest/netboot/mini.iso
- drop to shell and confirm that calling `wget` runs GNU wget.
- witness eventual failure retrieving packages (bsdutils is the first to fail every time. syslog below.)

When I went back and did the same thing with a stock version of wget-udeb added, the install worked. Again, I dropped to shell and confirmed it was busybox responding to `wget`.

This doesn't seem right. We know that debootstrap works fine with GNU wget. That's probably the most common way it's used. The other strange part is that only some of the packages fail to download; many others are retrieved just fine.

[Mark Russell (marrusl)]

All the packages that fail have colons (":") in their version strings. precise for example:

$ rmadison bsdutils busybox-initramfs diffutils dmsetup iputils-ping libattr1 libdevmapper1.02.1 libgcc1 libusb-0.1-4 login ntpdate passwd procps vim-common vim-tiny zlib1g | grep precise
bsdutils | 1:2.20.1-1ubuntu3 | precise | amd64, armel, armhf, i386, powerpc
busybox-initramfs | 1:1.18.5-1ubuntu4 | precise | amd64, armel, armhf, i386, powerpc
busybox-initramfs | 1:1.18.5-1ubuntu4.1 | precise-updates | amd64, armel, armhf, i386, powerpc
diffutils | 1:3.2-1ubuntu1 | precise | source, amd64, armel, armhf, i386, powerpc
dmsetup | 2:1.02.48-4ubuntu7 | precise | amd64, armel, armhf, i386, powerpc
dmsetup | 2:1.02.48-4ubuntu7.3 | precise-updates | amd64, armel, armhf, i386, powerpc
iputils-ping | 3:20101006-1ubuntu1 | precise | amd64, armel, armhf, i386, powerpc
libattr1 | 1:2.4.46-5ubuntu1 | precise | amd64, armel, armhf, i386, powerpc
libdevmapper1.02.1 | 2:1.02.48-4ubuntu7 | precise | amd64, armel, armhf, i386, powerpc
libdevmapper1.02.1 | 2:1.02.48-4ubuntu7.3 | precise-updates | amd64, armel, armhf, i386, powerpc
libgcc1 | 1:4.6.3-1ubuntu5 | precise | amd64, armel, armhf, i386, powerpc
libusb-0.1-4 | 2:0.1.12-20 | precise | amd64, armel, armhf, i386, powerpc
login | 1:4.1.4.2+svn3283-3ubuntu5 | precise | amd64, armel, armhf, i386, powerpc
login | 1:4.1.4.2+svn3283-3ubuntu5.1 | precise-updates | amd64, armel, armhf, i386, powerpc
ntpdate | 1:4.2.6.p3+dfsg-1ubuntu3 | precise | amd64, armel, armhf, i386, powerpc
ntpdate | 1:4.2.6.p3+dfsg-1ubuntu3.1 | precise-updates | amd64, armel, armhf, i386, powerpc
passwd | 1:4.1.4.2+svn3283-3ubuntu5 | precise | amd64, armel, armhf, i386, powerpc
passwd | 1:4.1.4.2+svn3283-3ubuntu5.1 | precise-updates | amd64, armel, armhf, i386, powerpc
procps | 1:3.2.8-11ubuntu6 | precise | source, amd64, armel, armhf, i386, powerpc
vim-common | 2:7.3.429-2ubuntu2 | precise | amd64, armel, armhf, i386, powerpc
vim-common | 2:7.3.429-2ubuntu2.1 | precise-updates | amd64, armel, armhf, i386, powerpc
vim-tiny | 2:7.3.429-2ubuntu2 | precise | amd64, armel, armhf, i386, powerpc
vim-tiny | 2:7.3.429-2ubuntu2.1 | precise-updates | amd64, armel, armhf, i386, powerpc
zlib1g | 1:1.2.3.4.dfsg-3ubuntu4 | precise | amd64, armel, armhf, i386, powerpc

[Dimitri John Ledkov (xnox)]

Colon in the package version string denotes epoch, and is not present in the actual .deb filename.
For example, just on a normal system:
$ apt-get download bsdutils
Get:1 Downloading bsdutils 1:2.20.1-5.1ubuntu8 [40.4 kB]
Fetched 40.4 kB in 1s (32.8 kB/s)
$ ls
bsdutils_2.20.1-5.1ubuntu8_amd64.deb

But this does suggest that something in the installer relies on exact string matching, somewhere...

A simple test between GNU and busybox, does show differences in the output:
$ wget http://127.0.0.1/:
--2013-07-09 10:45:24-- http://127.0.0.1/:
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-09 10:45:24 ERROR 404: Not Found.

$ busybox wget http://127.0.0.1/:
Connecting to 127.0.0.1 (127.0.0.1:80)
wget: server returned error: HTTP/1.1 404 Not Found

Now looking into debian-installer-utils, reveals README.wget404 which explains in detail that fetching urls, depends on text matching of wget's output.

Have you considered adding a new fetch-url-methods/https, which uses wget.gnu (if available) with correct wget.gnu 404 parsing?
Or one can modify existing fetch-url-methods/http to accommodate for either busybox or GNU.

[Colin Watson (cjwatson)]

Make fetch-url-methods/http cope with either, please.

[Adam Conrad (adconrad)]

It gets more interesting than the above analysis, because we're calling wget with -q. In busybox, this still prints errors, in GNU wget, this shuts it up completely. The equivalent mode in GNU wget would be --no-verbose, but busybox doesn't provide that switch.

[Dimitri John Ledkov (xnox)]

Unsubscribing ubuntu-sponsors for the moment, as proposed patches cause installer regressions.

[Colin Watson (cjwatson)]

Oh! I just spotted the problem here. When running in d-i, debootstrap uses base-installer/pkgdetails.c to provide progress output. That looks for anything vaguely like a percentage, which matches the URL-encoded colon ("%3a").

I still have some work to do for this, but it has my attention now that I've figured out that particularly inscrutable failure.

[Colin Watson (cjwatson)]

There's a matching perl implementation in debootstrap which should technically also be adjusted.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-installer - 1.122ubuntu21

---------------
base-installer (1.122ubuntu21) trusty; urgency=medium

  * Cherry-pick from trunk:
    - pkgdetails.c: Only interpret percentages following whitespace, to cope
      with GNU wget outputting the local file name (which may contain "%"
      due to URL-encoding) after it finishes the download (LP: #1172101).
 -- Colin Watson <email address hidden> Fri, 07 Feb 2014 16:38:27 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package debian-installer-utils - 1.103ubuntu1

---------------
debian-installer-utils (1.103ubuntu1) trusty; urgency=medium

  * Resynchronise with Debian. Remaining changes:
    - user-params: Don't propagate vga=*, break=*, *-ubiquity, or
      noninteractive to installed system.
    - list-devices:
      + New "mmc-partition" type for detecting partitions on MMC devices.
      + Make sure we also detect platform-omap MMCs.
      + Don't return entries for extended partitions.
    - Don't include the battery subsystem on calls to udevadm trigger.
  * Drop Ubuntu delta to use /proc/self/fd/4 rather than /dev/fd/4 in
    fetch-url-methods/http. udevd reliably creates the /dev/fd symlink
    these days.

debian-installer-utils (1.103) unstable; urgency=medium

  * fetch-url-methods/http: Cope with the slightly different no-such-file
    output produced by GNU wget, and with it needing to be invoked using
    --no-verbose rather than -q (LP: #1172101).
  * Merge from Ubuntu:
    - log-output: Always install a no-op SIGCHLD handler, in case the
      subsidiary process starts a daemon which does not fully disconnect its
      standard file descriptors (LP: #1021293). See also the changelog for
      1.46.

debian-installer-utils (1.102) unstable; urgency=low

  [ Updated translations ]
  * Bosnian (bs.po) by Amila Valjevčić

debian-installer-utils (1.101) unstable; urgency=low

  [ Updated translations ]
  * Ukrainian (uk.po) by Yuri Chornoivan
 -- Colin Watson <email address hidden> Fri, 07 Feb 2014 17:26:21 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package wget - 1.15-1ubuntu1

---------------
wget (1.15-1ubuntu1) trusty; urgency=medium

  [ Colin Watson ]
  * Resynchronise with Debian. Remaining changes:
    - Add wget-udeb to ship wget.gnu as alternative to busybox wget.
    - Build-depend on libssl-dev 0.9.8k-7ubuntu4.
    - Pass --with-ssl=openssl; there's no udeb for gnutls.
    - Add a second build pass for the udeb, so we can build with -Os and
      without libidn.
    - Use dh_autotools-dev instead of custom config.{sub,guess} copy.

  [ Mark Russell ]
  * debian/rules: build wget-udeb to install its binary as /usr/bin/wget
    instead of /usr/bin/wget.gnu (LP: #1172101).

wget (1.15-1) unstable; urgency=medium

  * new upstream release from 2014-01-19
      Wget: fails with long file names in URLs Closes: #672131
      Wget omits Host header for CONNECT Closes: #699337
      Wget: Inaccurate catalan translation Closes: #697081
      Cannot write to ... (Success) Closes: #716938
      Regression: write error on wget -c for already fully retrieved file
      Closes: #696700
      wget: NTLM not supported Closes: #718262
      wget --no-check-certificate does check certificate in certain conditions
      Closes: #686837
  * debian/control updated Standard-Version; no changes needed

wget (1.14.96.38327-2) experimental; urgency=low

  * debian/rules fix configure option --with-libidn Closes: #728735

wget (1.14.96.38327-1) experimental; urgency=low

  * 1.15 alpha version from 2013-11-02
    - removed patches which are included now upstream:
      wget-doc-fixitemx2item.patch
      wget-doc-remove2.nv.patch
      wget-doc-texi2pod_fixperl5.18change.patch
    - included fixes for
      Wget: fails with long file names in URLs Closes: #672131
      Wget omits Host header for CONNECT Closes: #699337
      Wget: Inaccurate catalan translation Closes: #697081
      Cannot write to ... (Success) Closes: #716938
      Regression: write error on wget -c for already fully retrieved file
      Closes: #696700
      wget: NTLM not supported Closes: #718262
      wget --no-check-certificate does check certificate in certain conditions
      Closes: #686837
  * debian/control add Recommends ca-certificates to get
    https URLs working. Closes: #712540
  * debian/rules fix lintian warning dh-clean-k-is-deprecated
 -- Colin Watson <email address hidden> Fri, 07 Feb 2014 17:42:45 +0000

[Mark Russell (marrusl)]

Hi Colin,

Thanks so much for the updates. Interesting to see that debootstrap was was actually involved. I did want to clarify a couple of things about the cert-checking side of the issue. But I think that should go in LP bug 833994.

Thanks!

[Colin Watson (cjwatson)]

Yes, I haven't got to the cert-checking bit yet; that's for next week.

[Launchpad Janitor (janitor)]

This bug was fixed in the package debootstrap - 1.0.57

---------------
debootstrap (1.0.57) unstable; urgency=medium

  * pkgdetails_perl: Only interpret percentages following whitespace, to
    cope with GNU wget outputting the local file name (which may contain "%"
    due to URL-encoding) after it finishes the download (LP: #1172101).

 -- Colin Watson <email address hidden> Fri, 07 Feb 2014 16:12:23 +0000

[Dave Chiluk (chiluk)]

cjwatson, what are the chances of getting these changes pushed into precise?

As for cert-checking, I don't think that's as necessary as the people who want to enable https archives, are typically pointing at internal mirrors that contain invalid certificates.

[Colin Watson (cjwatson)]

I'm not comfortable with enabling this unless it includes a viable cert-checking story. Fortunately I think I have one; see my comments in bug 833994.

I think we can probably SRU this to precise, yes. However, I want to get it all done end-to-end (including both HTTPS preseeding and package acquisition over HTTPS) in trusty first, otherwise there'll be a lot of confusion during verification as people observe that it's only half-complete.

[Chris J Arges (arges)]

Hello Mark, or anyone else affected,

Accepted base-installer into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/base-installer/1.122ubuntu7.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Mark, or anyone else affected,

Accepted debian-installer-utils into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/debian-installer-utils/1.88ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Mark, or anyone else affected,

Accepted debootstrap into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/debootstrap/1.0.40~ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Mark, or anyone else affected,

Accepted wget into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/wget/1.13.4-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mark Russell (marrusl)]

Confirmed. This is fixed. Thanks.

[Colin Watson (cjwatson)]

The verification of the Stable Release Update for wget has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-installer - 1.122ubuntu7.4

---------------
base-installer (1.122ubuntu7.4) precise; urgency=medium

  * pkgdetails.c: Only interpret percentages following whitespace, to cope
    with GNU wget outputting the local file name (which may contain "%" due
    to URL-encoding) after it finishes the download (LP: #1172101).
  * Add HTTPS support: pass --no-check-certificate to debootstrap if
    debian-installer/allow_unauthenticated_ssl=true, set https_proxy if
    necessary, and copy any certificates that exist in d-i into the target
    system (LP: #1135163).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:13:11 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package debootstrap - 1.0.40~ubuntu0.6

---------------
debootstrap (1.0.40~ubuntu0.6) precise; urgency=medium

  * pkgdetails_perl: Only interpret percentages following whitespace, to
    cope with GNU wget outputting the local file name (which may contain "%"
    due to URL-encoding) after it finishes the download (LP: #1172101).
  * Install apt-transport-https and ca-certificates if installing from an
    HTTPS mirror (LP: #1135163).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:40:02 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package wget - 1.13.4-2ubuntu1.1

---------------
wget (1.13.4-2ubuntu1.1) precise; urgency=medium

  [ Mark Russell ]
  * debian/rules: build wget-udeb to install its binary as /usr/bin/wget
    instead of /usr/bin/wget.gnu (LP: #1172101).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:43:44 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package debian-installer-utils - 1.88ubuntu2.2

---------------
debian-installer-utils (1.88ubuntu2.2) precise; urgency=medium

  * fetch-url-methods/http: Cope with the slightly different no-such-file
    output produced by GNU wget, and with it needing to be invoked using
    --no-verbose rather than -q (LP: #1172101).
  * Add HTTPS support to fetch-url, which will only work if d-i has been
    built with GNU wget; debian-installer/allow_unauthenticated_ssl implies
    the --no-check-certificate option (LP: #833994).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:35:11 +0100

[Poil (poil)]

Hi,

I've tried to update my installer from 12.04 to 12.04.5 and all packages referenced in the comment #10 failed to download. (All is working fine when I switch back on the 12.04 installer)

Is this fix is included in the 12.04.5 netboot (http://cdimage.ubuntu.com/netboot/12.04/) ?

Best regards,