[improvement] no easy way to add packages required by apt before apt stage

[Till Riedel]

I am running cloud-init on a libvirt/virsh with this image: https://cdimage.debian.org/cdimage/openstack/archive/9.6.5-20190122/debian-9.6.5-20190122-openstack-amd64.qcow2

This is not really a bug, because I believe it is rather a packaging problem of many distros. So please close this asap if you do not feel responsible ;)

This is why I also posted this problem on the debian-cloud mailing list, but maybe this can be fixed also in cloud-init.

If apt-transport-https or dirmngr are not installed (they are no dependencies of cloud-init).

apt:
  sources:
   docker:
     source: 'deb [arch=amd64] https://download.docker.com/linux/debian stretch stable'
     keyserver: keyserver.ubuntu.com
     keyid: 0EBFCD88

you will run into problems like:

[ 14.011339] cloud-init[469]: E: The method driver /usr/lib/apt/methods/https could not be found.
[ 14.013524] cloud-init[469]: Cloud-init v. 0.7.9 running 'modules:config' at Wed, 23 Jan 2019 16:28:06 +0000. Up 10.01 seconds.
[ 14.016921] cloud-init[469]: 2019-01-23 16:28:10,921 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed

and

[ 14.460293] cloud-init[448]: Command: ['gpg', '--keyserver', 'keyserver.ubuntu.com', '--recv', '0EBFCD88']
[ 14.463206] cloud-init[448]: Stderr: gpg: failed to start the dirmngr '/usr/bin/dirmngr': No such file or directory

Unfortunatedly I could not find a way to call anything to add those packages before apt is run so that I had to patch the images.

One solution would be to allow multistage apt install in one way or the other (I know it adds complexity), so that one could install the relevant packages before the package list is modified and updated. Actually checking if a line contains https or a keyid and installing the appropriate dependencies could also work.

 Probably it would be much easier to fix the package dependency on cloud-init itself for the relevant distros. The downside of this is that it adds a bunch of packages as further dependencies, so I guess distros are not really willing to do that...

Thanks in advance,

Till

PS: I can live with having written some scripts patching the images after download...

[do3meli (d-info-e)]

i ran in to the exact same problem. i see it the same way and believe we have to add dirmngr as package dependency for cloud-init.

i found the following mailing list for debian cloud images that discusses this issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910654

the link references a merge request with more details on it: https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/22

basically they will/have added gnupg has a hard dependency on dirmngr for debian buster. so most likely they will fix it in the image itself.

for older images like debian stretch and as a fallback logic i think we should adjust cloud-init to always install dirmngr before running apt configuration module.

@Scott Moser: Any idea how to achieve this?

[do3meli (d-info-e)]

btw: i have fixed this with a little workaround in the cloud-init config by running a bootcmd that installs the package very early in the boot process:

bootcmd:
    - apt -y install dirmngr

[Julian Andres Klode (juliank)]

Just some words from the APT side:

So apt-transport-https is irrelevant for 18.04+ and buster, it's just a transitional package, and it _should not_ be installed on those.

Regarding key management: keyserver.ubuntu.com talks https, it's trivial to get keys from it without involving the gpg crap. Not sure what the others do, but I'd generally advise against the use of other keyservers. Heck, I'd prefer serving a .asc or .gpg file and having a keyurl to dump that file into trusted.gpg.d

[Julian Andres Klode (juliank)]

So in fact, HKP is quite easy to do by hand, e.g.:

http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x99242560

where 0x99242560 is the keyid field and keyserver.ubuntu.com is the keyserver.

It's trivial to do that yourself in cloud-init, there's no need to involve all the gnupg and dirmngr mess.

[Julian Andres Klode (juliank)]

That said, if you do, do require HKPS (HKP using HTTPS).

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3320