invalid sources.list generated from reading apt-setup/local0/repository

[Filippo Giunchedi]

[Impact]
Server users installing with a preseeded local repository.

[Test case]
Preseed "apt-setup/local0/repository" for a server install.
The installation process should complete normally, and the local repository should be enabled on the installed system.

[Regression Potential]
This adds support for handling local repositories and configuring them in preseed without adding the "deb" prefix to specify the type of source entry. There is no likely regression type except that of enabling local repositories on installs where the preseed is present and currently seen as invalid by base-installer where the issue was not noticed by systems administrators.

---

hi,
it looks like configure_apt from library.sh will add the contents of apt-setup/local0/repository verbatim to the sources file. This works only if the repository is prefixed with "deb ", which for apt-setup is optional. Not using a repository prefixed with "deb" will result in an invalid sources.list file. See also https://phabricator.wikimedia.org/T94177 and https://gerrit.wikimedia.org/r/250380

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paul Weaver (paul-weaver-uk)]

I can confirm this bug, which means the preseed file at
https://help.ubuntu.com/16.04/installation-guide/example-preseed.txt

# Additional repositories, local[0-9] available
#d-i apt-setup/local0/repository string \
# http://local.server/ubuntu xenial main
#d-i apt-setup/local0/comment string local server
# Enable deb-src lines
#d-i apt-setup/local0/source boolean true
# URL to the public key of the local repository; you must provide a key or
# apt will complain about the unauthenticated repository and so the
# sources.list line will be left commented out
#d-i apt-setup/local0/key string http://local.server/key

Doesn't work.

If you change the line to
d-i apt-setup/local0/repository string deb http://local.server....

The install works fine

I don't think that library.sh is necessarily the cause, as the above preseed works fine in 14.04 with the same code and behaviour from library.sh (it creates a sources.list file with a missing deb on the front). I don't understand

Under 1404 the following happens

Feb 29 14:02:41 base-installer: E: Type 'http://myserver/debs' is not known on line 4 in source list /etc/apt/sources.list
Feb 29 14:02:41 base-installer: E: The list of sources could not be read.
Feb 29 14:02:41 base-installer: warning: apt update failed: 100

However base-installer progresses anyway.

The function in library.sh that deals with local0 (and not local1-9) seems to be a bit of a hack though, but for anyone running into the same problem prepending deb to your preseed file should do the trick.

[kokarn (kokarn)]

Any progress on this?

[Lars Kollstedt (lk-x)]

Hi all,

I also stumbled into this Bug in Ubuntu 16.4 LTS (xenial).

I can't really confirm this for ubuntu 14.4 (trusty) so far. I can confirm the the line causing this to be already in 1.144ubuntu1 but it doe'snt seem to have effect (at least for me and our pxe install environment).

My last 14.4 installation ist about 3 weeks ago, now. But I added the workarrounds, when starting our tests for Xenial-Deploment. From the dates I've seen, this should be already in use on 14.4 (even with a netboot initrd, created in march 2015) but perhaps the file created here is never used in some cases, because it is already overwritten or overridden by another udeb (e.g. apt-setup(-udeb) itself).

The appended patch should fix this, but keep the workarrounds working. ;-)

All lines above in library.sh function configure_apt are containing the leading deb and since apt-setup ignores and adds the deb here this should IMHO be also done here. ;-) The way to do this was borrowed from apt-setup generators/60-local.

Kind regards,
    Lars

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "MissingDeb-2016-05-02.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Colin Watson (cjwatson)]

Lars, this patch actually breaks it for both forms :-) But I'll fix it up and commit, thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-installer - 1.158ubuntu3

---------------
base-installer (1.158ubuntu3) yakkety; urgency=medium

  * Tolerate omission of leading "deb " in apt-setup/local0/repository
    (based on patch by Lars Kollstedt; LP: #1512347).

 -- Colin Watson <email address hidden> Wed, 04 May 2016 00:43:55 +0100

[Lars Kollstedt (lk-x)]

Hi Collin,

oops, missed the deb again. :-( ;-) Thank you for correcting. :-) :-)

Attached the corrected one for completeness. ;-)

base-installer (1.158ubuntu3) already contains the corrected. Obviously you have also corrected the indentation matching the lines above in 1.158ubuntu3. :-)

As far as I can see 1.158ubuntu3 is only availabe for Yakkety Yak (active development) at the moment, hope it will get into Xenial Xerus as a bugfix within the next weeks also. ;-)

Most people affected by this would be LTS users.

Thank you. :-)

Kind regards,
    Lars

[Adam Conrad (adconrad)]

Hello Filippo, or anyone else affected,

Accepted base-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/base-installer/1.158ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Schlomo Schapiro (sschapiro)]

Can you please create installation media which includes this package? Or provide instructions how to test this fix which actually applies to the preseed installation and not to a running system?

[Colin Watson (cjwatson)]

@sschapiro: You can test this by using a 16.04 netboot image (http://cdimage.ubuntu.com/netboot/), and putting apt-setup/proposed=true on the installer's kernel command line or adding "d-i apt-setup/proposed boolean true" to your preseed file. (The result will be that the installed system has unverified updates on it too, so it would be best to only use this for testing this fix.)

[Lars Kollstedt (lk-x)]

Hi Adam, hi Schlomo,

for me it works with the package that has been put into proposed on the
evening of May 11th 2016.

Sorry for the late reply.

On Wednesday, 11. Mai 2016, 16:42:12 Adam Conrad wrote:
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Your feedback will aid us getting this update
> out to other Ubuntu users.

https://wiki.ubuntu.com/Testing/EnableProposed:
[...]
|
| Installation testing using -proposed
|----------------------------------------------------------
|
| Sometimes you may be asked to test a netboot installer image from trusty-
| proposed. The images may be found here (replace "i386" with your
| architecture as necessary):
|
| http://archive.ubuntu.com/ubuntu/dists/trusty-proposed/main/installer-i386/
|
| In order to install successfully from these images, you will normally need
| to tell the installer to fetch its own components from -proposed as well,
| which is not the default. To do this, add the following boot parameter:
|
| apt-setup/proposed=true
|
| You must make sure that the mirror you are installing from contains packages
| from -proposed. All official mirrors will do so; if you operate your own
| mirror or use a site-local mirror, it may need to be modified to pull from
| -proposed. Make sure that you do not simply use a loopback mount of a CD or
| DVD image as an installation source; this is one of the cases where such
| mirrors will not work, since they do not contain the updated kernel packages
| required by the new installer.
|
|
[...]

Tests I've done:
 1. First I compared the versions between archive.ubuntu.com and our mirror,
    to make sure the Cronjob running at night took over all the changes, and
    the upstream mirror also did.
 2. In the second step I added the "proposed" option to our PXE-Environment,
    to the preseed-File, and to our automation script creating and updating
    the directory structure for PXE (including the initrd with the) and the
    Customized Netboot-ISO-Image.
 3. Then I removed the "deb" workaround from our preseed-File, and called the
    automation script to put our new preeseed-File into initrd.
    It is now containing:
     - the apt-setup/proposed=true
     - and the local0/repository *without* the manually prepended "deb"
 4. Now I installed the testing machine with proposed. The installation was
    successfully done, this failed before (with 1.158ubuntu2).
    All selected packages were installed, the proposed-Repository was also
    included after installation and "debconf-get-selections --installer" shows
    the local0/repository without the prepended deb.

    This looks fine, now.

 5. Then I added the Workarround "deb" again, and called the
    automation script to put our new preeseed-File into
    initrd.
    It is now containing:
     - the apt-setup/proposed=true
     - and the local0/repository *with* the manually prepended "deb"
       Workarround.
 6. Now I installed the testing machine with proposed, again. Installation was
    also successful.
    All selected packages were installed, the proposed-Reposito...

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-installer - 1.158ubuntu2.1

---------------
base-installer (1.158ubuntu2.1) xenial; urgency=medium

  * Tolerate omission of leading "deb " in apt-setup/local0/repository
    (based on patch by Lars Kollstedt; LP: #1512347).

 -- Mathieu Trudel-Lapierre <email address hidden> Wed, 04 May 2016 09:14:44 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for base-installer has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.