KeyError: 'ServiceName' during snapdevicegw prod deploy

[Jonathan Hartley]

When snapstore team were deploying snapdevicegw to prod today, the deploy stopped with an exception (shown at bottom of this text)

The failure had no bad consequences as far as I know. I just had to retry the deploy command and it then finished without error. I'm reporting this because I want our deploys to finish reliably, without training the snap store team to ignore errors during prod deploys.

Just in case, the deploy command in question was:

    ACTIVE=f30c7aa ./service-do snapdevicegw make manifests/deploy-switch-service

The error traceback,
found in wendigo /srv/mojo/mojo-prod-ols-snap-store/xenial/production/log/mojo.log:

2019-11-05 21:03:12 [DEBUG] deployer.env: Delta unit: prometheus/0 change:{'since': '2019-11-05T21:03:12.272434824Z', 'current': 'active', 'version': '', 'message': 'Ready'}
Traceback (most recent call last):
  File "/usr/bin/juju-deployer", line 9, in <module>
    load_entry_point('juju-deployer==0.10.0', 'console_scripts', 'juju-deployer')()
  File "/usr/lib/python3/dist-packages/deployer/cli.py", line 140, in main
    run()
  File "/usr/lib/python3/dist-packages/deployer/cli.py", line 250, in run
    importer.Importer(env, deployment, options).run()
  File "/usr/lib/python3/dist-packages/deployer/action/importer.py", line 306, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python3/dist-packages/deployer/action/importer.py", line 277, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python3/dist-packages/deployer/env/go.py", line 279, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python3/dist-packages/jujuclient/watch.py", line 166, in run
    callback(*change)
  File "/usr/lib/python3/dist-packages/deployer/env/go.py", line 292, in _delta_event_log
    name = self._format_endpoints(d['Endpoints'])
  File "/usr/lib/python3/dist-packages/deployer/env/go.py", line 308, in _format_endpoints
    eps[0]['ServiceName'],
KeyError: 'ServiceName'

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 109, in apport_excepthook
    pr.add_proc_info(extraenv=['PYTHONPATH', 'PYTHONHOME'])
  File "/usr/lib/python3/dist-packages/apport/report.py", line 513, in add_proc_info
    self.add_proc_environ(pid, extraenv)
  File "/usr/lib/python3/dist-packages/apport/report.py", line 580, in add_proc_environ
    env = _read_file('environ', dir_fd=proc_pid_fd).replace('\n', '\\n')
  File "/usr/lib/python3/dist-packages/apport/report.py", line 73, in _read_file
    with open(path, 'rb', opener=lambda path, mode: os.open(path, mode, dir_fd=dir_fd)) as fd:
  File "/usr/lib/python3/dist-packages/apport/report.py", line 73, in <lambda>
    with open(path, 'rb', opener=lambda path, mode: os.open(path, mode, dir_fd=dir_fd)) as fd:
TypeError: argument should be integer, not list

[Jonathan Hartley (tartley)]

I pestered Tim Van Steenburgh about this, and he helpfully got me up to speed:

> "My team hasn't supported/maintained/worked on juju-deployer since mid-2017. By that time, many
> features had been merged into juju itself, and it was decided to direct any new stuff into the
> python juju client (https://github.com/juju/python-libjuju)."