apt-key failure is fatal

[Francis Ginther]

Hit this from a automated system test run (https://ci.lscape.net/job/landscape-system-tests/2461/) trying to deploy a Landscape Openstack Autopilot cloud (OBJECT=ceph, BLOCK=iscsi, LDS=16.05~bzr10071~release43+jenkins131-1):

[From /var/log/juju/unit-glance-simplestreams-sync-0.log]
2016-05-19 08:53:43 INFO install gpg: keyring `/tmp/tmpymwhfvuw/secring.gpg' created
2016-05-19 08:53:43 INFO install gpg: keyring `/tmp/tmpymwhfvuw/pubring.gpg' created
2016-05-19 08:53:43 INFO install gpg: requesting key 4652B4E6 from hkp server keyserver.ubuntu.com
2016-05-19 08:53:44 INFO install gpgkeys: key 35F77D63B5CEC106C577ED856E85A86E4652B4E6 can't be retrieved
2016-05-19 08:53:44 INFO install gpg: no valid OpenPGP data found.
2016-05-19 08:53:44 INFO install gpg: Total number processed: 0
2016-05-19 08:53:44 INFO install Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.r3WjgmYPPD --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv 4652B4E6
2016-05-19 08:53:44 INFO install gpg: requesting key 4652B4E6 from hkp server keyserver.ubuntu.com
2016-05-19 08:53:44 INFO install gpgkeys: key 4652B4E6 can't be retrieved
2016-05-19 08:53:44 INFO install gpg: no valid OpenPGP data found.
2016-05-19 08:53:44 INFO install gpg: Total number processed: 0
2016-05-19 08:53:44 INFO install Traceback (most recent call last):
2016-05-19 08:53:44 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install", line 281, in <module>
2016-05-19 08:53:44 INFO install hooks.execute(sys.argv)
2016-05-19 08:53:44 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2016-05-19 08:53:44 INFO install self._hooks[hook_name]()
2016-05-19 08:53:44 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/install", line 199, in install
2016-05-19 08:53:44 INFO install add_source(hookenv.config('source'), hookenv.config('key'))
2016-05-19 08:53:44 INFO install File "/var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/hooks/charmhelpers/fetch/__init__.py", line 315, in add_source
2016-05-19 08:53:44 INFO install key])
2016-05-19 08:53:44 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2016-05-19 08:53:44 INFO install raise CalledProcessError(retcode, cmd)
2016-05-19 08:53:44 INFO install subprocess.CalledProcessError: Command '['apt-key', 'adv', '--keyserver', 'hkp://keyserver.ubuntu.com:80', '--recv', u'4652B4E6']' returned non-zero exit status 2
2016-05-19 08:53:44 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

The failure to retrieve the key requires networking and can be a transient failure. Should this be retried? A single failure is currently fatal to the charms install hook and causes the Autopilot deploy to fail.

[Francis Ginther (fginther)]

Attaching /var/log/juju/unit-glance-simplestreams-sync-0.log.

[James Page (james-page)]

if this is a transient failure, I think a few retries might be appropriate; marking triaged and 'low' for now.

[Ursula Junque (ursinha)]

This problem could happen in all charms (as it's a charm-helpers bug), but all the last four times it manifested we could see it in gs3 logs. I'll create an autopilot-log-analyser issue for that and if it happens in another charm, we can extend it to catch that.

[Francis Ginther (fginther)]

Another flavor of this (at least in that apt-key failed and led to an eventual charm install failure). This one from https://ci.lscape.net/job/landscape-system-tests/5125.

[from landscape-0-inner-logs/ceilometer-0/var/log/juju/unit-landscape-client-57.log]
2017-01-28 10:08:04 INFO install gpg: keyring `/tmp/tmp9p2uy5f6/secring.gpg' created
2017-01-28 10:08:04 INFO install gpg: keyring `/tmp/tmp9p2uy5f6/pubring.gpg' created
2017-01-28 10:08:04 INFO install gpg: requesting key 4652B4E6 from hkp server keyserver.ubuntu.com
2017-01-28 10:08:14 INFO install gpgkeys: key 35F77D63B5CEC106C577ED856E85A86E4652B4E6 can't be retrieved
2017-01-28 10:08:14 INFO install gpg: no valid OpenPGP data found.
2017-01-28 10:08:14 INFO install gpg: Total number processed: 0
2017-01-28 10:08:14 INFO install gpg: keyserver communications error: keyserver helper general error
2017-01-28 10:08:14 INFO install gpg: keyserver communications error: unknown pubkey algorithm
2017-01-28 10:08:14 INFO install gpg: keyserver receive failed: unknown pubkey algorithm
2017-01-28 10:08:14 INFO install Ign:1 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton InRele
ase
2017-01-28 10:08:14 INFO install Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
2017-01-28 10:08:14 INFO install Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
2017-01-28 10:08:14 INFO install Get:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Releas
e [7881 B]
2017-01-28 10:08:14 INFO install Get:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
2017-01-28 10:08:15 INFO install Get:6 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/newton Releas
e.gpg [543 B]
2017-01-28 10:08:15 INFO install Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
2017-01-28 10:08:15 INFO install Get:8 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xe
nial InRelease [17.6 kB]
2017-01-28 10:08:15 INFO install Get:9 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [202 kB
]
017-01-28 10:08:16 INFO install Ign:8 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease
2017-01-28 10:08:16 INFO install Get:10 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial/main amd64 Packages [864 B]
2017-01-28 10:08:16 INFO install Get:11 http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial/main Translation-en [364 B]
2017-01-28 10:08:22 INFO install Fetched 536 kB in 2s (246 kB/s)
2017-01-28 10:08:24 INFO install Reading package lists...
2017-01-28 10:08:24 INFO install W: GPG error: http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6E85A86E4652B4E6
2017-01-28 10:08:24 INFO install W: The repository 'http://ppa.launchpad.net/landscape/landscape-client-cloud-installer/ubuntu xenial InRelease' is not signed.
2017-01-28 10:08:25 INFO juju-log Installing ['landscape-client'] with options: ['--option=Dpkg::Options::=--force-confold']
2017-01...