Solutions QA CI has seen several failures in the elasticsearch charms install hook when apt-key add is call from 'ansible-playbook -c local playbook.yaml --tags install'
The test run details are available here:
https://solutions.qa.canonical.com/#/qa/testRun/924ec573-2b16-4f00-8954-f3e69c89feb8
Below is the error output in unit-elasticsearch-0.log, found in the juju crashdump.
2020-02-25 19:41:22 DEBUG install TASK [Add an apt key in a raw format.] *****************************************
2020-02-25 19:41:26 DEBUG install fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/apt-key add -", "msg": "Warning: apt-key output should not be parsed (stdout is not a terminal)\ngpg: can't connect to the agent: IPC connect call failed", "rc": 2, "stderr": "Warning: apt-key output should not be parsed (stdout is not a terminal)\ngpg: can't connect to the agent: IPC connect call failed\n", "stderr_lines": ["Warning: apt-key output should not be parsed (stdout is not a terminal)", "gpg: can't connect to the agent: IPC connect call failed"], "stdout": "", "stdout_lines": []}
2020-02-25 19:41:26 DEBUG install to retry, use: --limit @/var/lib/juju/agents/unit-elasticsearch-0/charm/playbook.retry
2020-02-25 19:41:26 DEBUG install
2020-02-25 19:41:26 DEBUG install PLAY RECAP *********************************************************************
2020-02-25 19:41:26 DEBUG install localhost : ok=1 changed=0 unreachable=0 failed=1
2020-02-25 19:41:26 DEBUG install
2020-02-25 19:41:28 DEBUG install Traceback (most recent call last):
2020-02-25 19:41:28 DEBUG install File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/install", line 152, in <module>
2020-02-25 19:41:28 DEBUG install hooks.execute(sys.argv)
2020-02-25 19:41:28 DEBUG install File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 229, in execute
2020-02-25 19:41:28 DEBUG install self.playbook_path, tags=[hook_name], extra_vars=extra_vars)
2020-02-25 19:41:28 DEBUG install File "/var/lib/juju/agents/unit-elasticsearch-0/charm/hooks/charmhelpers/contrib/ansible/__init__.py", line 161, in apply_playbook
2020-02-25 19:41:28 DEBUG install subprocess.check_call(call, env=env)
2020-02-25 19:41:28 DEBUG install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-02-25 19:41:28 DEBUG install raise CalledProcessError(retcode, cmd)
2020-02-25 19:41:28 DEBUG install subprocess.CalledProcessError: Command '['ansible-playbook', '-c', 'local', 'playbook.yaml', '--tags', 'install']' returned non-zero exit status 2.
2020-02-25 19:41:28 ERROR juju.worker.uniter.operation runhook.go:132 hook "install" failed: exit status 1
I am not able to reproduce this issue in local juju env, the add_key task works smoothly for me.
By searching the error message:
gpg: can't connect to the agent: IPC connect call failed
the results show that it is normally because a gpg-agent is already running, but current process can not connect to it.
The suggested solution is to kill it and let it restart.
According to `man gpg-agent`:
1. The agent is automatically started on demand by gpg, gpgsm, gpgconf, or gpg-connect-agent. Thus there is no reason to start it manually.
2. If you want to manually terminate the currently-running agent, you can safely do so with: gpgconf --kill gpg-agent
So it's safe to kill the agent before apt_key, to avoid potential gpg connection issue. It will start on demand.
Add a patch to kill agent before apt_key tasks:
https:/
It may or may not fix this issue, but wouldn't be any harm.
Task tested without issue:
unit-es-0: 22:13:40 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:41 DEBUG unit.es/
unit-es-0: 22:13:44 DEBUG unit.es/