deploy hook fails due to "KeyError: 'username'" despite juju trust cmd

[Dan Ackerson]

Version Info
=============
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
$ lxc version
Client version: 3.14
Server version: 3.14
$ juju version
2.6.5-bionic-amd64
$ juju status openstack-integrator
Model Controller Cloud/Region Version SLA Timestamp
default lxd localhost/localhost 2.6.5 unsupported 13:31:02Z
App Version Status Scale Charm Store Rev OS Notes
openstack-integrator error 1 openstack-integrator jujucharms 22 ubuntu
Unit Workload Agent Machine Public address Ports Message
openstack-integrator/0* error idle 33 10.0.8.183 hook failed: "config-changed"
Machine State DNS Inst id Series AZ Message
33 started 10.0.8.183 juju-ccdbad-33 bionic Running

Presentation
=============
$ juju status
...
openstack-integrator/0* blocked executing 33 10.0.8.183 (config-changed) missing crede
ntials; grant with `juju trust` or set via config
...
$ juju trust openstack-integrator
$ juju status
...
openstack-integrator/0* error idle 33 10.0.8.183 hook failed: "config-changed"
...

Log info (juju-crashdump file attached)
=========
$ juju debug-log --replay --include openstack-integrator/0
unit-openstack-integrator-0: 13:15:56 DEBUG unit.openstack-integrator/0.juju-log tracer: hooks phase, 0 handlers queued
unit-openstack-integrator-0: 13:15:56 DEBUG unit.openstack-integrator/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++ queue handler reactive/openstack.py:17:update_creds
tracer: ++ queue handler reactive/openstack.py:30:get_creds
unit-openstack-integrator-0: 13:15:56 INFO unit.openstack-integrator/0.juju-log Invoking reactive handler: reactive/openstack.py:17:update_creds
unit-openstack-integrator-0: 13:15:56 INFO unit.openstack-integrator/0.juju-log Invoking reactive handler: reactive/openstack.py:30:get_creds
unit-openstack-integrator-0: 13:15:57 INFO unit.openstack-integrator/0.juju-log Checking credentials-get for credentials
unit-openstack-integrator-0: 13:15:57 ERROR unit.openstack-integrator/0.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/reactive/openstack.py", line 32, in get_creds
    toggle_flag('charm.openstack.creds.set', layer.openstack.get_credentials())
  File "lib/charms/layer/openstack.py", line 58, in get_credentials
    _merge_if_set(creds_data, _normalize_creds(_creds_data))
  File "lib/charms/layer/openstack.py", line 146, in _normalize_creds
    username=attrs['username'],
KeyError: 'username'

unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed Traceback (most recent call last):
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/hooks/config-changed", line 22, in <module>
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed main()
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed bus.dispatch(restricted=restricted_mode)
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed _invoke(other_handlers)
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed handler.invoke()
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed self._action(*args)
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/reactive/openstack.py", line 32, in get_creds
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed toggle_flag('charm.openstack.creds.set', layer.openstack.get_credentials())
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "lib/charms/layer/openstack.py", line 58, in get_credentials
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed _merge_if_set(creds_data, _normalize_creds(_creds_data))
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed File "lib/charms/layer/openstack.py", line 146, in _normalize_creds
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed username=attrs['username'],
unit-openstack-integrator-0: 13:15:57 DEBUG unit.openstack-integrator/0.config-changed KeyError: 'username'
unit-openstack-integrator-0: 13:15:57 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1

[Cory Johns (johnsca)]

We apparently don't handle the AccessKeyAuthType format for the creds returned by Juju: https://github.com/juju/juju/blob/develop/provider/openstack/credentials.go#L80-L104

This is going to impact the integrator charm, as well as anything that relates to it: https://github.com/juju-solutions/interface-openstack-integration/blob/master/requires.py#L88-L102

[Cory Johns (johnsca)]

It looks like the OpenStack provider in Kubernetes only supports username / password auth anyway: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/provider-configuration.md#global

I'm not sure that adding support for this auth method would actually help at this point and it sounds like the only option may be switching the Juju config to use username / password auth.

[Dan Ackerson (dan.ackerson)]

indeed, that's what shook it loose and finally worked:

$ vi creds.json
{
    "auth-url":"http://10.0.8.84:5000/v3",
    "username":"admin",
    "password":"xyz123",
    "project-name":"admin",
    "user-domain-name":"admin_domain",
    "project-domain-name":"admin_domain"
}

$ juju config openstack-integrator credentials="$(base64 ~/creds.json)"

[Daniel Niasoff (5-daniel-o)]

I had this issue and Dan's fix didn't work for me.

Solution was to update openstack cloud credentials for the controller and add the following fields

      user-domain-name: Default
      project-domain-name: Default

after that "juju trust openstack-integrator" worked

[Cory Johns (johnsca)]

PR: https://github.com/juju-solutions/charm-openstack-integrator/pull/36

Note: The original check for supported auth-type was added in PR#18: https://github.com/juju-solutions/charm-openstack-integrator/pull/18/files#diff-971956955be6e43ffa435d6402b78674R134-R136

[Cory Johns (johnsca)]

Cherry-pick to master: https://github.com/charmed-kubernetes/charm-openstack-integrator/commit/4e3afc60302e373a8c1a1ab2e6529b28d7be9a90