Xenial preseed fails to load key for 3rd party repo with apt-setup/local0/key

[Schlomo Schapiro]

I have an automated preseed installation that uses these lines to add custom repos during the installation:

d-i apt-setup/local0/repository string deb http://jschule.github.io/ubuntu/ /
d-i apt-setup/local0/comment string JTS local repository
d-i apt-setup/local0/source boolean false
d-i apt-setup/local0/key string http://jschule.github.io/ubuntu/repo.key

d-i apt-setup/local1/repository string deb http://dl.google.com/linux/chrome/deb/ stable main
d-i apt-setup/local1/comment string Google Chrome Browser
d-i apt-setup/local1/source boolean false
d-i apt-setup/local1/key string http://dl.google.com/linux/linux_signing_key.pub

(seehttps://github.com/jschule/ubuntu/blob/d46f1cef49ed71dc4bfe317119cccd3f39097ef4/preseed/jts.txt for complete preseed file that causes the problem).

In xenial the installation fails because the GPG key for the local0 repo is not loaded into the system so that it can be used (see screenshot). Strangely, "chroot /target apt-key list" shows the key 9E62229E to be installed.

Just to be sure that there is no problem with my repo and key I started the Xenial live CD and installed my repo there manually. All works well. IMHO this shows that the problem is clearly related to the automated installation with preseed.

Maybe this is related to #1512347, that was the only thing I could find on Launchpad that is in the same area.

If you want to reproduce this then you can checkout the scripts from https://github.com/jschule/ubuntu/tree/gh-pages/qemu and run "./run.sh xenial" to start my installation.

I found a very ugly workaround by changing the apt-setup lines to this:

d-i apt-setup/local0/repository string deb http://archive.canonical.com/ubuntu trusty partner
d-i apt-setup/local0/source boolean false

d-i apt-setup/local1/repository string deb http://jschule.github.io/ubuntu/ /
d-i apt-setup/local1/comment string JTS local repository
d-i apt-setup/local1/source boolean false
d-i apt-setup/local1/key string http://jschule.github.io/ubuntu/repo.key

d-i apt-setup/local2/repository string deb http://dl.google.com/linux/chrome/deb/ stable main
d-i apt-setup/local2/comment string Google Chrome Browser
d-i apt-setup/local2/source boolean false
d-i apt-setup/local2/key string http://dl.google.com/linux/linux_signing_key.pub

I suppose that the workaround works because now the local0 repo is one where the signing key is already part of Ubuntu. I just hope that there is no package in the trusty partner repo that is not also in the xenial partner repo.

For me it is very important that you fix this bug before 16.04 is released so that I can continue to use Ubuntu with an automated setup.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin (linux-support)]

Check your repo key: https://juliank.wordpress.com/2016/03/15/clarifications-and-updates-on-apt-sha1/

[Schlomo Schapiro (sschapiro)]

Thanks for the link. It looks like a warning and not an error. Also, I don't understand how this issue would explain the observed behavior which is that any key specified for the local0 repo fails while using the same key and repo as local1 repo works.

Since my repo key works as local1 repo I think that the problem is with the installer and not with the repo key.

Do you have any idea how to debug this further?

[Julian Andres Klode (juliank)]

This does not seem to be related to the SHA1 deprecation in APT - https://wiki.debian.org/Teams/Apt/Sha1Removal - as this would not get to that stage / show a warning there.

[Schlomo Schapiro (sschapiro)]

Any ideas how to fix this? Or how to debug it further? Can anybody else confirm this problem?

[Lars Kollstedt (lk-x)]

Hi all,

first thanks to ~juliank, this lead me to an workaround for this in my case.

In our case netboot install failed with a "no suitable kernel found with your apt settings" (message text written down from memory), when our internal software repository was included to bootstrap our deployment environment.

Switching from the ncurses-installer to a shell showed up, that /target/etc/apt/sources.list contains only a invalid placeholder for the main repository, when this error occurs. From my memory this was xenial.invalid but might also have been debootstrap.invalid.

Replacing the signing key by one with SHA-2-256 solved this, then I stumbled into Bug #1512347 which was already mentioned above.
That IMHO means Bug #1553121 is definitely a SHA-1 issue. Because first I missed the lines
| personal-digest-preferences SHA256
| cert-digest-algo SHA256
| default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
in ~/.gnupg/gpg.conf (on a Machine with Ubuntu 12.04 LTS (precise)) and created key signed with SHA-1 again (as visible with pgpdump).
With this mistake the error still occurs. ;-)

As far as I know ~anders-kaseorg should be right in Bug #1556666. The keys are statically imported to the trusted-Keychain. The SHA-1 o signature isn't used for any verification in any apt mechanisms I know. For this reason the warning in the output of apt-get update should be more than enough.

IMHO this should at least be catched with a propper error message.

I didn't find the lines causing this, yet. The gpgv calls in the debootstrap Package file functions should work, at least from the output on a fully installed xenial system. Another place doing similar stuff I haven't found.

The SHA1 warnings/errors also affects the repositories on http://downloads.linux.hp.com, but they don't offically support Ubuntu 16.4 LTS (xenial), yet.

Kind regards
    Lars

[Lars Kollstedt (lk-x)]

Sorry, was the old name:
> The SHA1 warnings/errors also affects the repositories on http://downloads.linux.hp.com, but they don't offically support
> Ubuntu 16.4 LTS (xenial), yet.
Of course this is http://downloads.linux.hpe.com/, now. ;-)

[Lars Kollstedt (lk-x)]

Hi all,

to clarify somthing that might be missunderstood in my previous post.

Of course the hpe stuff is OT. What I meant to say with this, is that the list on https://wiki.debian.org/Teams/Apt/Sha1Removal is still incompletely reflecting the repositories affected, and will probably always be.

There are plenty of additional repositories. Workaround in some cases might be to inherit the repository by adding it after installation. "apt-get update" only displays a warning. We were already doing this because, not all our servers were from HPE, and this works so far with the mentioned warning. ;-)

But in some cases such workarrounds aren't wanted. Or people are simply deploying the repository URL via preseed and get weired errors, now.

At least the last case *should* be fixed, since this will burn a lot of time. IMHO for nothing!

As far as I can see, neither apt-setup and base-installer are dealing with the keys themselves the verification done in debootstrap (used by base-installer) doesn't look like there is anything done with the HASH-Algorithm. Both are dealing with apt-get, but:
| root@xenial-test2:/home/kollstedt# apt-get -o APT::Get::List-Cleanup=false update; echo "return code = $?"
[...]
| Ign:7 http://downloads.linux.hpe.com/SDR/repo/mcp/ubuntu trusty/current InRelease
| Hit:8 http://downloads.linux.hpe.com/SDR/repo/mcp/ubuntu trusty/current Release
| Reading package lists... Done
| W: http://downloads.linux.hpe.com/SDR/repo/mcp/ubuntu/dists/trusty/current/Release.gpg: Signature by key
| 882F7199B20F94BD7E3E690EFADD8D64B1275EA3 uses weak digest algorithm (SHA1)
| return code = 0
Used the HPE for test, since our internal reposiory has a SHA-2-256 now.

I don't see them dealing with any output. ;-)

Any ideas so far?

Kind regards
   Lars

[Lars Kollstedt (lk-x)]

On Monday, 2. May 2016, 16:13:37 I wrote:
> As far as I know ~anders-kaseorg should be right in Bug #1556666. The
> keys are statically imported to the trusted-Keychain. The SHA-1 o
> signature isn't used for any verification in any apt mechanisms I know.
> For this reason the warning in the output of apt-get update should be
> more than enough.

On Monday, 2. May 2016, 18:14:00 I wrote:
> | W:
[...]
> | Signature by key 882F7199B20F94BD7E3E690EFADD8D64B1275EA3 uses
> | weak digest algorithm (SHA1)

On Monday, 2. May 2016, 18:14:00 I wrote:
> IMHO for nothing!

Hi all,

another correction to the backgrounds of the SHA-1 stuff. Someting missleads me
the SHA-1 signature on the key to be the security issue.
It's the signature in the Release.gpg / InRelease file that causes the issue.

For more details:
http://www.cs.umd.edu/~jkatz/papers/pgp-attack.pdf

Exchanging the Key would be usefull to make old signatures worthless for
future attacks. But the key itself is not / or at least should not be the
cause of this message. ;-)

That would mean that the lines in gpg.conf should probably fix that, if
exchanging the key is not an option. If you're doing this, you must hope
nobody can find old signatures of your key.

An hash collision atack (the length attack is a special case of this) isn't an
easy attack, but for distributing malware as security update via an
compromised or malvious mirror server this isn't impossible someone does.

For this it's reasonable to warn or disable this by default with an error
message.

Now, from background informations back to the bug...

Of course this does'nt change the following...

On Monday, 2. May 2016, 16:13:37 I wrote:
> In our case netboot install failed with a "no suitable kernel found with
> your apt settings" (message text written down from memory), when our
> internal software repository was included to bootstrap our deployment
> environment.
[...]
> IMHO this should at least be catched with a propper error message.

On Monday, 2. May 2016, 18:14:00 I wrote:
> Or people are simply
> deploying the repository URL via preseed and get weired errors, now.
>
> At least the last case *should* be fixed, since this will burn a lot of
> time.

And I didn't mean my time, I know this Bug, now. ;-) :-D

As mentioned in my previos messages, I tried to find the lines causing this but
without success, or at least without being able to reproduce this with the
eqations of the binarys in an usual xenial installation (the scripts would
IMHO run with the udeb eqations of gpgv and apt-get - I asummed they're
behaving equally but do they?).

This is'nt really easy to trobleshoot because there are mutiple special
packages for installation and bootstraping playing together. ;-)

Kind regards,
 Lars

--
man-da.de GmbH, AS8365 Phone: [phone]
Mornewegstraße 30 Fax: [phone]
D-64293 Darmstadt e-mail: <email address hidden>
Geschäftsführer Marcus Stögbauer AG Darmstadt, HRB 94 84

[Lawren Quigley-Jones (lquigley)]

The SHA1 vs SHA256 is an issue but I don't believe it's coming into play with this bug. I did have to change my signing process but now I'm signing my Release.gpg with SHA256 and I'm still unable to add a local repo via `d-i apt-setup/local0/repository`.

I install local packages during installation using `d-i pkgsel/include` so the netboot installation fails with the following error:
WARNING: The following packages cannot be authenticated!

It appears to me that the key import occurs after the verification but I might be missing something:

Aug 10 17:09:36 base-installer: Get:17 http://apt.local.server.com/apt ./ Packages [54.6 kB]
Aug 10 17:09:36 base-installer: Fetched 1494 kB in 2s (500 kB/s)
Aug 10 17:09:36 base-installer: Reading package lists...
Aug 10 17:09:37 base-installer:
Aug 10 17:09:37 base-installer: W
Aug 10 17:09:37 base-installer: :
Aug 10 17:09:37 base-installer: GPG error: http://apt.local.server.com/apt ./ Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 1234567890ABCDEFG
Aug 10 17:09:37 base-installer:
Aug 10 17:09:37 base-installer: W
Aug 10 17:09:37 base-installer: :
Aug 10 17:09:37 base-installer: The repository 'http://apt.local.server.com/apt ./ Release' is not signed.
Aug 10 17:09:37 base-installer:
Aug 10 17:09:37 base-installer: W
Aug 10 17:09:37 base-installer: :
Aug 10 17:09:37 base-installer: There is no public key available for the following key IDs:
Aug 10 17:09:37 base-installer: 1234567890ABCDEFG
Aug 10 17:09:37 base-installer:
[...]
Aug 10 17:17:28 main-menu[239]: (process:23053): 2016-08-10 17:17:15 URL:http://apt.local.server.com/server.com.key [1185/1185] -> "/target/tmp/key0.pub" [1]
Aug 10 17:17:28 main-menu[239]: (process:23053): OK

I can install my local packages if I `chroot /target`. All I have to do is edit my /etc/apt/sources.list and comment out my local0 repo and `apt-get update` and then uncomment it and `apt-get update` again.

At this point the md5's have been imported however this gets done and my packages in my local repo install without a hitch. Based on this behavior it seems like the installer is skipping a step when it imports the Release file for local0.

I can verify that I am able to see my key when I `apt-key list` both before and after my `apt-get update`.

I can confirm that setting local0 to xenial main and using local1 for my local repo does bypass this bug. I can also confirm that this all works in trusty.

I hope this is useful.

Thanks

[catsem (csc03)]

I can confirm this bug.
You can reconstruct with puppetlabs PC1 repo which has a key including sha-2 hashes.

d-i apt-setup/local0/key string http://apt.puppetlabs.com/pubkey.gpg
d-i apt-setup/local0/repository string http://apt.puppetlabs.com xenial PC1
d-i apt-setup/local0/comment string Puppetlabs PC1 xenial Repository
d-i apt-setup/local0/source boolean true

Who the hell broke the preseed part of Xenial installer?... this is only one bug of many many others
With Trusty preseed was working quite well...

I'm asking myself why I'm commenting this bug... seems like nobody really cares. This issue has been added 2016-03-04...

Xenial is still not ready for business use, even in .1 release...

[catsem (csc03)]

Xenial is still not ready for business use, even in .2 release...

[Andy Newton (anewton)]

Just hit this same problem adding the Puppetlabs PC1 repo. As far as I can tell, the key is deployed, but apt hasn't been updated to use it.

Installation halts at the "select packages" phase. At this point, if I chroot into /target, the Puppet Labs GPG key is present but if I try to install puppet-agent, it shows as unauthenticated.

If I run apt-get update and THEN try the install, puppet-agent is installed OK.

Can confirm that the workaround still works (make PC1 the local1 repo).

[Markus Wigge (markus-cultcom)]

I can confirm that bionic is affected too.

Maybe the installer image is missing some gnupg tools? I observed that with a minimal install I cannot use "apt-key" within the installed system because there is no gnupg installed at all.

This is really disappointing.

[Lars Kollstedt (lk-x)]

Hi Markus,

this was originally a Bug about SHA-1 signed repositories in local0 breaking the entire installation. The Error messages displayed, where/are not really good, changing to the shell of the install system, and having a look at /var/log/syslog there shows more.

This was in 2016 and mainly a topic for xenial. But if you are still using SHA1 signed repositories this will affect you in some way in any newer release, too. Bionic removes any repositories that have invalid keys, whereas xenial fails to install the base system because all repositories where not loaded.

The missing gnupg tools are affecting bionic, I would wonder if this will affect artful, too. For xenial this is definitely working.
The missing gnupg tools are discussed in Bug #1754075 for bionic. There are two ways to fix. As far as I know none of them has been applied, yet.

This bug is not duplicated and IMHO not really solved, but it has lost importance since SHA-1 signed repositories are getting rarer.

Kind regards
   Lars

[Sonny (aadityabhatia)]

I can confirm that this bug affects cosmic as well. :(