APT repo PGP keys not handled appropriately

[Darren R. Starr]

For further reference within this bug, I provide the example file https://pastebin.ubuntu.com/p/txzYWpY33j/

I am attempting to employ the apt tools provided within the cloud-init system to configure a new apt repository for installing docker during initial deployment. This is of course preferred to using shell commands as this would ideally allow better handling of issues of package installation and also hopefully be more secure.

packages:
...
  - docker-ce
  - docker-ce-cli
  - containerd.io

Are described within the user-data and

apt:
  sources:
    docker:
      source: "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu focal stable"
      key: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
...
        -----END PGP PUBLIC KEY BLOCK-----

Is provided as the corresponding key. This key is obtained by running the command :

curl -fsSL https://download.docker.com/linux/ubuntu/gpg

I am using the stock Ubuntu 20.04.2 cloud image as the guest platform.

The expected/desired behavior of this would be to dearmor the GPG key provided into the /usr/share/keyrings directory with a filename (sourcename)-archive.gpg

The actual behavior however is to install the GPG key within the /etc/trusted.gpg directory which appears to no longer be supported as apt-key is no longer supported (my reference for this information is https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html)

As such, apt (update|upgrade|install) does not operate correctly and the following is seen in /var/log/cloud-init-output.log

W: GPG error: https://download.docker.com/linux/ubuntu focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
E: The repository 'https://download.docker.com/linux/ubuntu focal InRelease' is not signed.
2021-05-31 13:40:10,144 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed

[James Falcon (falcojr)]

Thanks for the detailed bug report.

You can make your example "work" by removing the "signed-by=/usr/share/keyrings/docker-archive-keyring.gpg" section on your deb line, though this is insecure for the reasons you linked.

cloud-init is currently adding raw keys via the apt-key command. We'll need to change this.

[Paride Legovini (paride)]

Hi Darren,

I think there's some confusion here, and hopefully this is not-a-bug. You are probably trying to translate the "Install Docker Engine on Ubuntu" instructions [1] in cloud-init terms. The instructions basically tell you to:

1. Install the (dearmoured) Docker repository gpg key as /usr/share/keyrings/docker-archive-keyring.gpg
2. Add an apt source which uses that key to verify the package signatures. This is done using the sources.list "signed-by=" parameter.

When you add a repository via cloud-init the key you specify is added to the default location for trusted repository keys, so you should drop the signed-by=/usr/share/keyrings/docker-archive-keyring.gpg option from the 'source' line. This is basically what James suggested, what I'm adding is that it's not a workaround: it's working as intended, and the signatures are verified.

I'm setting the status of this report to Incomplete for now, leaving it open for further discussion. If you agree with my analysis please change its status to Invalid, otherwise comment back and we'll look at it again. Thanks!

[1] https://docs.docker.com/engine/install/ubuntu/

[Paride Legovini (paride)]

(cloud-init does indeed use apt-key behind the scenes, and the tool is deprecated, but it's still working as expected and will be available up to Ubuntu 22.04.)

[Darren R. Starr (drstarr)]

Hi Paride,

Thanks so much (to you and James) for putting in the effort.

I have just been through the source code to apt-add-repository (https://tracker.debian.org/pkg/software-properties) and from what I can tell, there's no reason why the code in that repo shouldn't extend the "software-properties" library to properly provide a home for gpg keys and I also believe that apt-add-repository should be appropriately modified to permit specifying keys. I believe this is the correct solution to the problem as I don't believe there should be hundreds of different implementations to adding and removing keys from the apt repos.

I've been in contact with the apt tools maintainers regarding the possibility of making an official patch to the apt tools to support key management. We discussed multiple different solutions. From what juliank in #debian-apt on irc.debian.org says :

- He thinks the best solution will eventually be to add keys directly to the source.d files in the /etc/apt configurations. This way, keys will be stored with the sources rather than being linked.

- He (who is Ubuntu's apt tool contributor it seems) will not have the time in his schedule in the near future.

- We agreed that making the CLI tools able to support this functionality would be a good thing. But would require someone to take the time to do it.

- He says that he thinks cloud-init should come up with their own solution to fit the cloud-init model to handle this. He wasn't being cocky or rude. He seems to just believe that this is something cloud-init should support through its own tools as well.

Based on the last point, I think that he means that it would make sense that when someone specifies a key within the user-data and ties it to a repo, then the key should be dearmored and stored in the correct location and the signed-by should be configured appropriately.

The main reason I'm on board with this solution is that the "software-properties" toolkit seems to be unmaintained at this time. And there's no official API for handling this. In theory it seems that the location for storing the keys is distribution specific and unless someone does start maintaining the software-properties tools again, it might be best to pretend like the only solution is to just write it yourself.

As I have a work-around at this time, I will ignore it for now. But since apt-key is scheduled to be yanked from distros in 2022, I'm sure this is a topic that will need to be revisited. This means either deprecating the key: tag from the cloud-init apt module side (and documenting that run-cmd is the proper solution)... or it would mean implementing the tools for replacing apt-key.

Anyway thank you so much for your support with this and I'll leave it up "to the experts" (meaning your group) to decide what is best to do.

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]