cannot bootstrap from repositories without InRelease files when --debian-installer set

[David McBride]

Version: 1.0.78+nmu1ubuntu1

I am using debootstrap in my own minimal system bootstrapper, and am making use of the progress information reported to FH 3 as enabled by the --debian-installer command-line flag.

Unfortunately, using this flag causes the behaviour of debootstrap to change, specifically in its handling of InRelease / Release files.

During normal operation, debootstrap will first attempt to fetch an InRelease file from the repository; if this is unavailable, i.e. the fetch fails with 404, then debootstrap will normally fall back to fetching a Release file instead. Indeed, this is the behaviour of debootstrap if --debian-installer is not passed on the command-line.

debootstrap uses the wgetprogress() function to fetch these URLs. It reads:

wgetprogress () {
        [ ! "$VERBOSE" ] && QSWITCH="-q"
        local ret=0
        if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
                wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW $PROGRESS_NEXT $PROGRESS_END >&3
                ret=$?
        else
                wget $QSWITCH "$@"
                ret=$?
        fi
        return $ret
}

When the --debian-installer command-line flag is set, the first path of the if branch will be taken - running the output of wget through a pipe, so that the $PKGDETAILS command can be used to parse progress information provided by wget and report it to FH 3 in a format usable by debian-installer.

However, if the URL passed to wget returns 404, then while the wget command
will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not, and the wgetprocess () function thus erroneously returns success.

As a consequence, debootstrap does not fall back to fetching and using a
Release file as it should in this case, and the bootstrapping attempt as a
whole fails.

Because this is a POSIX shell-script, I don't believe there is a
straightforward mechanism to fetch the exit status of the wget command when it is part of a pipeline. (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11). It might be possible to enhance the code invoked by $PKGDETAILS to return a fatal exit status if it does not definitely see a successful file retrieval?

A more direct work-around is to modify this function to unconditionally use the second code path that does not attempt to invoke wget as part of a pipeline. However, this does mean that you lose out on intra-file download progress reporting.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jacob Doherty (theflatnz)]

I believe we may be striking this bug using crouton and running Ubuntu in a chroot.
See: https://github.com/dnschneid/crouton/issues/2966

For us it's likely upstream as crouton uses the latest debootstrap release (not the ubuntu package) but guessing both are affected and hoping the ubuntu package maintainers might be able to verify and raise with their upstream.

The symptoms users are reporting are that the InRelease file does not exist for precise/trusty, it will re-try several times and then stop - seemingly not checking for a Release file.

[Sebastian Unger (sebunger44)]

I'm also affected by this. The attached patch fixes the problem using only posix shell mechanisms.

Please note that the paths in the diff are absolute to patch the problem in-system.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "functions.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Sebastian Unger (sebunger44)]

It looks as though this has been fixed in Ubuntu somewhere between Bionic and Eoan. Shouldn't this bug be updated accordingly?