cisco repo unusable after cobbler install

[Chris Ricker]

cobbler adds the havana repo to /etc/apt/sources.list:

deb http://openstack-repo.cisco.com/openstack/cisco havana-proposed main

but does not add the key for it:

# apt-key list
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <email address hidden>
sub 2048g/79164387 2004-09-12

pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <email address hidden>

pub 4096R/C0B21F32 2012-05-11
uid Ubuntu Archive Automatic Signing Key (2012) <email address hidden>

pub 4096R/EFE21092 2012-05-11
uid Ubuntu CD Image Automatic Signing Key (2012) <email address hidden>
#

As a result, apt is left with a corrupted partially unverified cache

To fix, you have to

- add the key manually

echo '-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)

mQENBE/oXVkBCACcjAcV7lRGskECEHovgZ6a2robpBroQBW+tJds7B+qn/DslOAN
1hm0UuGQsi8pNzHDE29FMO3yOhmkenDd1V/T6tHNXqhHvf55nL6anlzwMmq3syIS
uqVjeMMXbZ4d+Rh0K/rI4TyRbUiI2DDLP+6wYeh1pTPwrleHm5FXBMDbU/OZ5vKZ
67j99GaARYxHp8W/be8KRSoV9wU1WXr4+GA6K7ENe2A8PT+jH79Sr4kF4uKC3VxD
BF5Z0yaLqr+1V2pHU3AfmybOCmoPYviOqpwj3FQ2PhtObLs+hq7zCviDTX2IxHBb
Q3mGsD8wS9uyZcHN77maAzZlL5G794DEr1NLABEBAAG0NU9wZW5TdGFja0BDaXNj
byBBUFQgcmVwbyA8b3BlbnN0YWNrLWJ1aWxkZEBjaXNjby5jb20+iQE4BBMBAgAi
BQJP6F1ZAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDozGcFPtOxmXcK
B/9WvQrBwxmIMV2M+VMBhQqtipvJeDX2Uv34Ytpsg2jldl0TS8XheGlUNZ5djxDy
u3X0hKwRLeOppV09GVO3wGizNCV1EJjqQbCMkq6VSJjD1B/6Tg+3M/XmNaKHK3Op
zSi+35OQ6xXc38DUOrigaCZUU40nGQeYUMRYzI+d3pPlNd0+nLndrE4rNNFB91dM
BTeoyQMWd6tpTwz5MAi+I11tCIQAPCSG1qR52R3bog/0PlJzilxjkdShl1Cj0RmX
7bHIMD66uC1FKCpbRaiPR8XmTPLv29ZTk1ABBzoynZyFDfliRwQi6TS20TuEj+ZH
xq/T6MM6+rpdBVz62ek6/KBcuQENBE/oXVkBCACgzyyGvvHLx7g/Rpys1WdevYMH
THBS24RMaDHqg7H7xe0fFzmiblWjV8V4Yy+heLLV5nTYBQLS43MFvFbnFvB3ygDI
IdVjLVDXcPfcp+Np2PE8cJuDEE4seGU26UoJ2pPK/IHbnmGWYwXJBbik9YepD61c
NJ5XMzMYI5z9/YNupeJoy8/8uxdxI/B66PL9QN8wKBk5js2OX8TtEjmEZSrZrIuM
rVVXRU/1m732lhIyVVws4StRkpG+D15Dp98yDGjbCRREzZPeKHpvO/Uhn23hVyHe
PIc+bu1mXMQ+N/3UjXtfUg27hmmgBDAjxUeSb1moFpeqLys2AAY+yXiHDv57ABEB
AAGJAR8EGAECAAkFAk/oXVkCGwwACgkQ6MxnBT7TsZng+AgAnFogD90f3ByTVlNp
Sb+HHd/cPqZ83RB9XUxRRnkIQmOozUjw8nq8I8eTT4t0Sa8G9q1fl14tXIJ9szzz
BUIYyda/RYZszL9rHhucSfFIkpnp7ddfE9NDlnZUvavnnyRsWpIZa6hJq8hQEp92
IQBF6R7wOws0A0oUmME25Rzam9qVbywOh9ZQvzYPpFaEmmjpCRDxJLB1DYu8lnC4
h1jP1GXFUIQDbcznrR2MQDy5fNt678HcIqMwVp2CJz/2jrZlbSKfMckdpbiWNns/
xKyLYs5m34d4a0it6wsMem3YCefSYBjyLGSd/kCI/CgOdGN1ZY1HSdLmmjiDkQPQ
UcXHbA==
=v6jg
-----END PGP PUBLIC KEY BLOCK-----' | apt-key add -

- then blow away the cached apt indices with:

apt-get clean
cd /var/lib/apt
rm -rf lists
mkdir -p lists/partial
apt-get clean
apt-get update

Without the key added, apt-get update will error with:

W: GPG error: http://openstack-repo.cisco.com havana-proposed Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY E8CC67053ED3B199

Once the key is added, without the manual cache clearing, any apt-get update will error with:

W: GPG error: http://openstack-repo.cisco.com havana-proposed Release: The following signatures were invalid: BADSIG E8CC67053ED3B199 OpenStack@Cisco APT repo <email address hidden>

perhaps related is after cobbler and puppet are done we have duplicate entries between /etc/apt/sources.list and /etc/apt/sources.list.d/cisco* so get these warnings:

W: Duplicate sources.list entry http://openstack-repo.cisco.com/openstack/cisco/ havana-proposed/main amd64 Packages (/var/lib/apt/lists/openstack-repo.cisco.com_openstack_cisco_dists_havana-proposed_main_binary-amd64_Packages)
W: Duplicate sources.list entry http://openstack-repo.cisco.com/openstack/cisco/ havana-proposed/main i386 Packages (/var/lib/apt/lists/openstack-repo.cisco.com_openstack_cisco_dists_havana-proposed_main_binary-i386_Packages)
W: You may want to run apt-get update to correct these problems

(really a separate issue but I suspect fixing the cobbler will fix the above too)

[Mark T. Voelker (mvoelker)]

So there are a couple of things to look at here....I think basically what we want to do is:

1.) In the preseed, deliver the key as well. It's currently allowed to load packages unauthenticated which works, but has obvious implications.
2.) In the preseed, figure out a way to get the repo set up in /etc/apt/sources.list.d/... instead of sources.list. That makes the puppet run later basically a no-op. I'd rather not remove the puppet code that adds the repo as that allows the repo to be set up on preconfigured nodes (e.g. nodes not provisioned by cobbler).

Alternately, we can change the puppet code to put the repo info in sources.list. That might theoretically be a bit easier since preseed is basically a travesty when it comes to flexibility and documentation. =)

If we do want to put the key in preseed: AFAIK you can't include a key directly in a preseed, but instead have to include a URL from which it can be fetched (I need to verify this). That will be ugly in no-net situations, so we'll probably want to provision it to the build node's http server as part of the build node profile, then serve it from there.

[Mark T. Voelker (mvoelker)]

One further note on this: it looks like adding the repo via "d-i apt-setup" is going to be problematic in general. The only way to add a key there that I can see is to set a URL from which the key should be downloaded--but that request will go through any proxy that is configured via "d-i mirror/http/proxy" apparently, which for COI cases is ACNG on the build node. ACNG won't permit this (though we might be able to make it work with some tinkering). We might have to resort to just doing it all manually in late_command...which might not be completely awful since it would make it slightly easier to deal with the repo duplication issue.

[Mark T. Voelker (mvoelker)]

PR's:
https://github.com/CiscoSystems/puppet-coi/pull/31
https://github.com/CiscoSystems/puppet-cobbler/pull/59