Deprecated tool apt-key is still used for repo gpg keys

[Christian Rohmann]

When adding external package repos on Ubuntu / Debian the apt-key tool or rather the Ansible apt_key module [1] is used by various Openstack-Ansible roles. Examples are [3], [4], [5], [6] and there are likely more.

Debian / Ubuntu deprecated the apt-key tool for a while now - see [2]. While the tool is still there it will go away eventually. Also running `apt update` does show an warning every time:

```
# apt update
[...]
Hit:1 https://downloads.mariadb.com/MariaDB/mariadb-10.6.16/repo/ubuntu jammy InRelease
[...]
Hit:10 https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy InRelease
Hit:11 https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy InRelease
Fetched 229 kB in 2s (140 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: http://downloads.mariadb.com/MariaDB/mariadb-10.6.16/repo/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
```

Ansible does document an easy fix at [7] which simply is about downloading the key to `/etc/apt/keyrings/` and then using `apt_repository` to reference this file.

[1] https://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_key_module.html#notes
[2] https://manpages.debian.org/testing/apt/apt-key.8.en.html#DEPRECATION
[3] https://github.com/openstack/openstack-ansible-galera_server/blob/229ae217c1e127979cd5370414e6d2b13c910055/tasks/galera_install_apt.yml#L29
[4] https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/83d8ec2a3049e81c8cdb07acc97f16ab2d57e08e/tasks/install_apt.yml#L33
[5] https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/83d8ec2a3049e81c8cdb07acc97f16ab2d57e08e/tasks/install_apt.yml#L33
[6] https://github.com/openstack/openstack-ansible-ceph_client/blob/43c9071f4f3acc827fd079ba0823bcbd4052b5e2/tasks/ceph_preinstall_apt.yml#L35
[7] https://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_key_module.html#examples

[Dmitriy Rabotyagov (noonedeadpunk)]

Full list of repos that still use apt-key can be found here: https://t.ly/EJZ4j

[Dmitriy Rabotyagov (noonedeadpunk)]

Also change does not look that trivial, given now we need to implement quite complex logic in order not to change user-facing interface.

If look into ceph_client role, we expect quite wide range of potential inputs from a user:
https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/tasks/ceph_preinstall_apt.yml#L36-L40

And there will be quite some code to handle cases where data/file/url is provided, as each are separate tasks with logic.

So I assume we either need to create a common role to handle that in one place, or own version of module that will take same input as apt-key, just manage keyrings without the binary.

[Christian Rohmann (christian-rohmann)]

Firstly thank you for picking up on this issue so quickly.

I did not mean to trivialize the issue and yes, a shared role or some set of reusable tasks might make sense to replace the use of apt_key.

But in regards to example at https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/tasks/ceph_preinstall_apt.yml#L36-L40 I don't understand why there can be no "breaking" changes? All the users of the roles care about is for it to manage the external repo (and the required signature keys) right?

If you read [1] the recommendation is to download / put the key for each repository in an individual file and then reference this in the apt source via "Signed-By" to avoid trusting the key for more than just the repo it belongs to. No keyrings or anything to update via gpg. And that's what all the upstream projects do provide [2], [3], [4] (the repo setup tool does download [5]).

[1] https://manpages.debian.org/testing/apt/apt-key.8.en.html#DEPRECATION
[2] Ceph: https://docs.ceph.com/en/quincy/install/get-packages/#apt
[3] RabbitMQ https://www.rabbitmq.com/install-debian.html#apt-quick-start-cloudsmith
[4] MariaDB: https://mariadb.com/docs/server/ref/repo/cli/mariadb_repo_setup/#Installation
[5] https://supplychain.mariadb.com/mariadb-keyring-2019.gpg

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, not really. There are umber of users with isolated from internet environments, and it's common to maintain own set of repositories with their own GPG keys in such cases.

So logic of where to take the key and how to place it to the destination is what adds some complications.

As today ceph_gpg_keys allows quite some flexibility on how to supply the key - either as a lookup and define data, or via URL, or as some different file on deploy host, and all should be still supported as a result.

[Dmitriy Rabotyagov (noonedeadpunk)]

Eventually following example of adding "Signed-By" to avoid trusting the key is even less trivial, because today we have independent and not related structures for GPG and repo installation.

So we definitely will need to somehow link repos to keys. And yes, this is potentially not doable without breaking changes :(

[Dmitriy Rabotyagov (noonedeadpunk)]

During the bug triage in IRC meeting it was proposed to leverage ansible.builtin.deb822_repository module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/deb822_repository_module.html

which should handle all required cases.

[Dmitriy Rabotyagov (noonedeadpunk)]

The following topic does track the progress for this bug:
https://review.opendev.org/q/topic:%22osa/apt_key%22

However, in order to use deb822 an apt 2.3.10 or higher is required, which is not available for Debian bulsseye which we can drop only for 2024.2 release.