hwpack-install asks for confirmation of not authenticated packages

[Alexander Sack]

when using hwpack-install i get asked like:

Do you want to continue [Y/n]? WARNING: The following packages cannot be authenticated!
  linux-image-linaro-vexpress u-boot-linaro-ca9x4-ct-vxp
Install these packages without verification [y/N]?

i think this is because we don't sign the Packages file in the hwpack. Doing that and whitelisting that
during install would probably help.

[James Westby (james-w)]

On Tue, 14 Sep 2010 19:32:54 -0000, Alexander Sack <email address hidden> wrote:
> Public bug reported:
>
> when using hwpack-install i get asked like:
>
> Do you want to continue [Y/n]? WARNING: The following packages cannot be authenticated!
> linux-image-linaro-vexpress u-boot-linaro-ca9x4-ct-vxp
> Install these packages without verification [y/N]?
>
> i think this is because we don't sign the Packages file in the hwpack. Doing that and whitelisting that
> during install would probably help.

Are you sure?

I haven't implemented putting the PPA keys in the hwpack yet, so it
might be that?

Let's make sure we understand the problem before trying to come up with
a fix.

Are you testing with the branch that favours downloading from the local
disk? Where do the packages come from if you choose yes?

Thanks,

James

[Guilherme Salgado (salgado)]

This happens for the debs that are included in the hwpack but I suspect it's because the hwpack doesn't include the gpg key used to sign the Packages file in it, or else it'd have been added to the apt keyring by linaro-hwpack-install.

[Steve Langasek (vorlon)]

On Wed, Sep 15, 2010 at 03:24:59PM -0000, Guilherme Salgado wrote:
> This happens for the debs that are included in the hwpack but I suspect
> it's because the hwpack doesn't include the gpg key used to sign the
> Packages file in it, or else it'd have been added to the apt keyring by
> linaro-hwpack-install.

Downloading unsigned hwpacks that contain a gpg for you to add to your apt
keyring is rather "security theater", though. Either the hwpacks themselves
should be signed and we should bypass the requirement for Packages signing
for the resulting local archive (similar to what's commonly done for CD
images), or the key used for Packages signing needs to be packaged and
included in the base images.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[James Westby (james-w)]

On Wed, 15 Sep 2010 17:58:21 -0000, Steve Langasek <email address hidden> wrote:
> On Wed, Sep 15, 2010 at 03:24:59PM -0000, Guilherme Salgado wrote:
> > This happens for the debs that are included in the hwpack but I suspect
> > it's because the hwpack doesn't include the gpg key used to sign the
> > Packages file in it, or else it'd have been added to the apt keyring by
> > linaro-hwpack-install.
>
> Downloading unsigned hwpacks that contain a gpg for you to add to your apt
> keyring is rather "security theater", though. Either the hwpacks themselves
> should be signed and we should bypass the requirement for Packages signing
> for the resulting local archive (similar to what's commonly done for CD
> images), or the key used for Packages signing needs to be packaged and
> included in the base images.

That's a good point.

There are two things going on here:

  * A hwpack may contain some debs, and currently they are indexed with
    an unsigned Packages file.

  * A hwpack may add sources to the system apt, and possibly download
    packages from there during operation, but they will remain after
    operation. Those may be signed, but you may not have the keys
    already.

If we sign the hwpacks, and distribute the public key for that in the
images then we could have some sort of trust path for both.

The current system is not at all set up for signing things though, and
I'm not sure if lexbuilder is much better.

Thanks,

James

[Marcin Juszkiewicz (hrw)]

"apt-get install -y --force-yes" or other way to tell apt-get that he should ignore lack of authorization will work I think.

[Guilherme Salgado (salgado)]

--force-yes is not what we want in the long term. We could use it as an interim solution, but maybe we should keep requiring authorization until this is fixed, so that we don't forget about it.

[Michael Hudson-Doyle (mwhudson)]

lexbuilder certainly has some support for signing things -- it's what makes files like http://snapshots.linaro.org/10.11-daily/linaro-headless/20101010/0/images/tar/linaro-m-headless-20101010-0.sha1sums.txt.asc. So we can probably sign the hwpacks somehow or other.

[James Westby (james-w)]

Alexander suggested that we could do the following for an interim solution

  * Copy the apt trustdb in to the chroot before running hwpack-install. This assumes that if the user trusts the repo on their own machine they trust it on the image too
  * Ship PPA keys that we use a lot (overlay, kernel, PPAs) with linaro-image-tools. We could either load them in to the machines trustdb, or have a custom trustdb that we copy over as well. If the user installs via a package then we have a trust path by those means.

This would eliminate the common causes, and we could tell engineers how to add extra keys to that for
PPAs that they use often.

It's easier than the signing solution in many ways, so could be a good interim solution, and may be good enough that we never go for signing.

What do people think?

Thanks,

James

[Loïc Minier (lool)]

What you describe sounds non-trivial; if we consider non-trivial, I'd like to propose something slightly different:
* sign hwpacks on snapshots.linaro.org/releases.linaro.org with some new key(s)
* ship these keys in linaro-image-tools and use them to verify hwpacks
* bundle keys of referenced repositories within hwpacks

The advantages are that we don't bundle anything specific to our hwpacks' contents in linaro-image-tools, and we also get hwpack signatures; this also allows people to list their own PPAs + signatures in their custom hwpacks. The disadvantage is some more work on the server side (keeping the signing keys).

[James Westby (james-w)]

On Fri, 18 Mar 2011 22:04:03 -0000, Loïc Minier <email address hidden> wrote:
> What you describe sounds non-trivial; if we consider non-trivial, I'd like to propose something slightly different:
> * sign hwpacks on snapshots.linaro.org/releases.linaro.org with some new key(s)
> * ship these keys in linaro-image-tools and use them to verify hwpacks
> * bundle keys of referenced repositories within hwpacks
>
> The advantages are that we don't bundle anything specific to our
> hwpacks' contents in linaro-image-tools, and we also get hwpack
> signatures; this also allows people to list their own PPAs + signatures
> in their custom hwpacks. The disadvantage is some more work on the
> server side (keeping the signing keys).

Right, that's exactly the more complex solution we were previously
discussing :-)

Thanks,

James

[Scott Bambrough (scottb)]

On Fri, 2011-03-18 at 21:01 +0000, James Westby wrote:
> Alexander suggested that we could do the following for an interim
> solution
>
> * Copy the apt trustdb in to the chroot before running hwpack-install. This assumes that if the user trusts the repo on their own machine they trust it on the image too
> * Ship PPA keys that we use a lot (overlay, kernel, PPAs) with linaro-image-tools. We could either load them in to the machines trustdb, or have a custom trustdb that we copy over as well. If the user installs via a package then we have a trust path by those means.
>
> This would eliminate the common causes, and we could tell engineers how to add extra keys to that for
> PPAs that they use often.
>
> It's easier than the signing solution in many ways, so could be a good
> interim solution, and may be good enough that we never go for signing.
>
> What do people think?

Could the PPA keys be part of the hardware pack itself?

Scott

--
Scott Bambrough <email address hidden>
Technical Director, Linaro Landing Teams

[Loïc Minier (lool)]

On Fri, Mar 18, 2011, James Westby wrote:
> Right, that's exactly the more complex solution we were previously
> discussing :-)

 ah right, sorry!

 To reword my opinion, the proposal from comment #8 seems to be a kludge
 90% as complex as the clean solution we all wish we had. ;-)

--
Loïc Minier

[James Westby (james-w)]

On Sat, 19 Mar 2011 00:11:56 -0000, Scott Bambrough <email address hidden> wrote:
> On Fri, 2011-03-18 at 21:01 +0000, James Westby wrote:
> > Alexander suggested that we could do the following for an interim
> > solution
> >
> > * Copy the apt trustdb in to the chroot before running hwpack-install. This assumes that if the user trusts the repo on their own machine they trust it on the image too
> > * Ship PPA keys that we use a lot (overlay, kernel, PPAs) with linaro-image-tools. We could either load them in to the machines trustdb, or have a custom trustdb that we copy over as well. If the user installs via a package then we have a trust path by those means.
> >
> > This would eliminate the common causes, and we could tell engineers how to add extra keys to that for
> > PPAs that they use often.
> >
> > It's easier than the signing solution in many ways, so could be a good
> > interim solution, and may be good enough that we never go for signing.
> >
> > What do people think?
>
> Could the PPA keys be part of the hardware pack itself?

That's the other solution that Loïc just described.

Thanks,

James

[James Westby (james-w)]

On Sat, 19 Mar 2011 00:18:55 -0000, Loïc Minier <email address hidden> wrote:
> On Fri, Mar 18, 2011, James Westby wrote:
> > Right, that's exactly the more complex solution we were previously
> > discussing :-)
>
> ah right, sorry!
>
> To reword my opinion, the proposal from comment #8 seems to be a kludge
> 90% as complex as the clean solution we all wish we had. ;-)

The difference is where the complexity is. My proposed solution is
entirely doable within code that we control, whereas the other requires
co-ordination in several places, which is why I think that it will be
easier to accomplish.

Thanks,

James

[Loïc Minier (lool)]

On Sun, Mar 20, 2011, James Westby wrote:
> The difference is where the complexity is. My proposed solution is
> entirely doable within code that we control, whereas the other requires
> co-ordination in several places, which is why I think that it will be
> easier to accomplish.

 We could cheat and have the signatures within the hwpacks, much like
 signed .debs.

--
Loïc Minier

[James Westby (james-w)]

On Sun, 20 Mar 2011 18:31:12 -0000, Loïc Minier <email address hidden> wrote:
> We could cheat and have the signatures within the hwpacks, much like
> signed .debs.

I'm not sure that makes the task significantly easier. We still have key
management to deal with.

Thanks,

James

[Alexander Sack (asac)]

I still prefer the approach of maintaining everything we need inside l-i-t code/package rather than something that is done on the server side. So I am +1 on #8

[Mattias Backman (mabac)]

Hi,

To me it seems like the solution in #8 is near trivial after all. I've just had a quick look at this so let me know if I'm oversimplifying.

> * Copy the apt trustdb in to the chroot before running hwpack-install. [...]

Sounds trivial, assuming that it just involves copying /etc/apt/trustdb.gpg to chroot_dir/apt/trustdb.gpg at the top of install_hwpacks.

> * Ship PPA keys that we use a lot (overlay, kernel, PPAs) with linaro-image-tools. We could either load them in to the machines trustdb, or have a custom trustdb that we copy over as well. [...]

Shipping separate keys would mean to just add them to the machines trustdb before the step above, I guess.

I could give it a go if you all think it's worth doing it this way?

Thanks,

Mattias

[James Westby (james-w)]

On Mon, 28 Mar 2011 12:35:07 -0000, Mattias Backman <email address hidden> wrote:
> Hi,
>
> To me it seems like the solution in #8 is near trivial after all. I've
> just had a quick look at this so let me know if I'm oversimplifying.
>
> > * Copy the apt trustdb in to the chroot before running hwpack-
> install. [...]
>
> Sounds trivial, assuming that it just involves copying
> /etc/apt/trustdb.gpg to chroot_dir/apt/trustdb.gpg at the top of
> install_hwpacks.

Yes, I think that's all it takes for this step. In addition, I think
that would be a useful thing to do anyway.

My only pause here would be that if we want to have extra keys for
hwpacks that aren't in the host system's trustdb we'll have to find some
way to combine them (two trustdbs, concatenating trustdbs, adding keys
later, or something)

I don't think that should stop us from making this simple change which
seems generally useful to me though.

We don't want to tie ourselves to using Debian-based systems as the host
any more than we already do, so make sure that it will still work if
there is no system trustdb.

> > * Ship PPA keys that we use a lot (overlay, kernel, PPAs) with
> linaro-image-tools. We could either load them in to the machines
> trustdb, or have a custom trustdb that we copy over as well. [...]
>
> Shipping separate keys would mean to just add them to the machines
> trustdb before the step above, I guess.

If we put them in the host system's trustdb, yes.

However, I'm not sure we would do that at runtime, we could have the
package do it at install time, as that would be quicker.

We could do it at runtime if we don't put them in the system
trustdb. This would likely be a loop of calls to

  sudo chroot $chroot apt-key add

at the top of install_hwpacks too. This would work on more than
Debian-based systems.

I'm not sure this is quite obviously the win that the first item
is. This is the part where there are multiple proposed solutions.

If you'll allow me to outline the other proposal in similar depth to
this one then we can compare the two better.

  * Sign the hwpacks
    - This is just a matter of calling GPG when building really, as
      offspring is already set up to use a key for signing results.
      - We may want to put it behind an option
    - We can choose to sign the hwpack internally or externally.
      - Externally means downloading an extra file and passing that
        extra file to l-m-c so that it can verify the signature.
      - Internally means signing something inside the hwpack. We want to
        sign the data that we are trusting. We could sign everything,
        but that would likely need a new hwpack format. (e.g. nested
        tarballs with the outer one containing a signature on the inner
        one)
        - I would argue that the files that are included are implicitly
          trusted here, and we can provide external signatures if
          someone wants to check that the hwpack did indeed come from
          us.
        - What we are specifically asserting is trusted here is a bunch
          of GPG public keys. We can't control what they are used to
          sign anyway, so it is merely an assertion that we...

[Loïc Minier (lool)]

From conversations on IRC, it seems this bug is different things to different people.

A couple of facts:
1) all Linaro images (rootfses and hwpacks) are already signed by a single GPG key which is per Offspring instance; there are separate .asc downloads for every image so that you can verify your download (if you have a trust path to this Offspring key).

2) while rootfses *might* currently include the PPA keys which were used to build the image (didn't check) the hwpacks do NOT include the PPA GPG keys of the repos used to build the hwpack.

I agree with James that it's quite some work to implement proper "builtin" (internal) signatures for hwpacks, it would be much less work to use the current external signatures (.asc). James is concerned that this makes an already complex process even more complex, but I think we would use our time better by making it easier to download Linaro bits (for instance by automation) rather than taking shortcuts on the security of hwpacks.

[Loïc Minier (lool)]

Here's an hopefully not too complex proposal which solves all issues I'm aware of:
* add the GPG key of the current Offspring instance to linaro-image-tools
* change linaro-media-create to verify rootfs.asc and hwpack.asc when passed --binary rootfs --hwpack hwpack args
* change linaro-hwpack-create to include PPA keys when the hwpack includes PPAs and linaro-hwpack-install to install them and setup the corresponding PPAs when the hwpack includes keys and PPAs
* provide a tool which downloads and verifies Linaro images (for instance, shell script or pygtk UI)

[James Westby (james-w)]

So, after further discussion with Loïc it's clear that we are currently discussing two different
issues in this bug

  * That the .debs included in the hwpack are not trusted by apt at hwpack-install time
  * That .debs downloaded from PPAs at hwpack-install time are not trusted

The bug is clearly about the former, and it is the larger issue being the common case, but
I was confused by early discussing in the life of hwpacks about the second issue. My comments
were about addressing that, and as such should be put to one side for now.

The issue is that apt inside the chroot at hwpack-install time can't check signatures of the
file:/// repo that we create due to it not being signed.

We have an immediate choice here:

  1) We could silence the warning without addressing the issue. This would mean that someone could Man In The Middle your download of the hwpack, and give you malicious content and it would be able to run code in a chroot on your host system (so able to do anything to your host), as well as in the target. This is pretty much true now, as most people won't actually checks the content. In addition, someone could MITM your image download, with one containing a malicious dpkg and get the same thing.

  2) We could address the warning by signing the file:/// repo we create (or equivalently get the assurances we would need to do so and then bypass apt's checks for that repo)

I think most will agree that 2 is what we should go for here. In order to do this we need assurance of origin (we need to know that the hwpack "came from Linaro"), so we need a signature verifiable on the client against a public key we know to be controlled by Linaro.

The easiest way to do this is to sign the entire hwpack externally. We already do this (the .sha1sums.txt.asc files constitute this), so we could use that. This would mean however that users have to download those files, and then pass them to linaro-media-create to have the signature checked.

We have a proposal to do a "testdrive" script that will automate downloading the needed pieces and creating an image. That will be able to do this easily, but we haven't scheduled work on that script yet.

[Mattias Backman (mabac)]

I'm just reading this from the bottom up now... The last suggestion from James could be just something like this: http://paste.ubuntu.com/590764/ with some checks along the way.

Could that be a part of the "testdrive" script?

I think this covers the last point from lool:
> * provide a tool which downloads and verifies Linaro images (for instance, shell script or pygtk UI)

but not these:
> * add the GPG key of the current Offspring instance to linaro-image-tools
> * change linaro-media-create to verify rootfs.asc and hwpack.asc when passed --binary rootfs --hwpack hwpack args
> * change linaro-hwpack-create to include PPA keys when the hwpack includes PPAs and linaro-hwpack-install to install them and setup the corresponding PPAs when the hwpack includes keys and PPAs

Would these still be needed even with the "outside" signing with the sha1sums files?

[James Westby (james-w)]

On Thu, 07 Apr 2011 13:38:54 -0000, Mattias Backman <email address hidden> wrote:
> I'm just reading this from the bottom up now... The last suggestion from
> James could be just something like this: http://paste.ubuntu.com/590764/
> with some checks along the way.
>
> Could that be a part of the "testdrive" script?

Yes, and I think that could be worth doing. I'm not particularly keen on
tying us to a second script that people need to use for things to work
well though.

> I think this covers the last point from lool:
> > * provide a tool which downloads and verifies Linaro images (for instance, shell script or pygtk UI)

Indeed.

> but not these:
> > * add the GPG key of the current Offspring instance to linaro-image-tools
> > * change linaro-media-create to verify rootfs.asc and hwpack.asc when passed --binary rootfs --hwpack hwpack args
> > * change linaro-hwpack-create to include PPA keys when the hwpack includes PPAs and linaro-hwpack-install to install them and setup the corresponding PPAs when the hwpack includes keys and PPAs

Indeed.

> Would these still be needed even with the "outside" signing with the
> sha1sums files?

Not really, but I would prefer a solution that worked with just
linaro-media-create. That's the only supported way to create a Linaro
image, so I'm fine with that.

I could be persuaded that for expediency we should just fix this
"externally."

Thanks,

James

[Mattias Backman (mabac)]

As a basis for further discussion, I made a prototype for
> * change linaro-media-create to verify rootfs.asc and hwpack.asc when passed --binary rootfs --hwpack hwpack args

Can you have a look at https://code.launchpad.net/~mabac/linaro-image-tools/bug-638384-hwpacks-signature/+merge/58648 ? I can't see why that part has to be more complicated than that. Needs checking that every hwpack actually is passed with a corresponding asc file of course and things like that.

Also needed is a way to make sure that the key for gpg verification is available.

[Mattias Backman (mabac)]

The option to pass asc files for binary and hwpacks has been merged and is included in 0.4.5. There is nothing forcing the use of it though.

Should perhaps the use of --hwpack-force-yes require that --hwpack-sig and --binary-sig options are passed?

[James Westby (james-w)]

On Wed, 27 Apr 2011 14:48:00 -0000, Mattias Backman <email address hidden> wrote:
> Should perhaps the use of --hwpack-force-yes require that --hwpack-sig
> and --binary-sig options are passed?

Nope, I think we want the force option to remain.

We do want to make --hwpack-force-yes unneeded if the options are passed
though. Either by doing what it already does, or by avoiding the
question from apt entirely.

Thanks,

James

[Mattias Backman (mabac)]

On Wed, Apr 27, 2011 at 5:18 PM, James Westby <email address hidden> wrote:
> On Wed, 27 Apr 2011 14:48:00 -0000, Mattias Backman <email address hidden> wrote:
>> Should perhaps the use of --hwpack-force-yes require that --hwpack-sig
>> and --binary-sig options are passed?
>
> Nope, I think we want the force option to remain.
>
> We do want to make --hwpack-force-yes unneeded if the options are passed
> though. Either by doing what it already does, or by avoiding the
> question from apt entirely.

The change in https://code.launchpad.net/~mabac/linaro-image-tools/bug-638384-hwpackforceyes-unneeded/+merge/59371
makes hwpacks install with --force-yes if the options are passed and
the hwpack integrity is confirmed. So the --hwpack-sig option is
optional but now benefits the user when passed.

Supplying a signature file for the rootfs is still optional and will
abort the script if verification fails.

[Dave Martin (dave-martin-arm)]

Just to note: this issue still applies in the 11.08 release deliverables.

[Dave Martin (dave-martin-arm)]

Maybe a silly point, but to simplify things, why not just display a disclaimer and prompt when l-m-c _starts_ explaining that the user should check the authenticity of l-m-c and the tarballs if they care about integrity.

If l-m-c, its dependencies and/or the rootfs tarball have been confirmed to be authentic, further authenticity checks are not meaningful.

An option could be supplied to allow frontends to suppress this behaviour (if, for example the frontend has confirmed authenticity by some other means)

[Alexander Sack (asac)]

thanks for putting this back on track, fathi.

mabac or someone else from infra, could you please summarize the latest and greatest solution you propose to move this forward?

[Mattias Backman (mabac)]

Since it's possible to supply the hwpack hashes to l-m-c to get rid of the warnings, I would think this is complete. The remaining question when I implemented this was how to assist the user so they don't have to download all the files themselves.

That piece has been implemented in linaro-fetch-image(-ui) where the user also gets assistance importing the keys if necessary.

Is there anything else that is needed?

[Mattias Backman (mabac)]

The fix in l-i-t was released almost a year ago. linaro-fetch-image takes care of the rest.