Initial status:
* openstack-integrator:client related to kubernetes-master:openstack and kubernetes-worker:openstack
Command issued:
* juju remove-application openstack-integrator
Expectation:
* openstack-integrator unit would get removed (this works fine)
* kubernetes-master would get reconfigured to work as before deploying openstack-integrator
* kubernetes-worker would get reconfigured to work as before deploying openstack-integrator (this seems to work fine. No errors are raised)
Problem:
* kubernetes-master's openstack-relation-departed hook fails with error:
unit-kubernetes-master-1: 12:57:10 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: reactive/kubernetes_master.py:1220:send_cluster_dns_detail
unit-kubernetes-master-1: 12:57:10 INFO unit.kubernetes-master/1.juju-log Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'get', 'service', '--namespace', 'kube-system', 'kube-dns', '--output', 'json']
unit-kubernetes-master-1: 12:57:10 INFO unit.kubernetes-master/1.juju-log Checking if snap.kube-apiserver.daemon is active (0 / 6)
unit-kubernetes-master-1: 12:57:10 INFO unit.kubernetes-master/1.juju-log Checking if snap.kube-controller-manager.daemon is active (0 / 6)
unit-kubernetes-master-1: 12:57:11 INFO unit.kubernetes-master/1.juju-log Checking if snap.kube-scheduler.daemon is active (0 / 6)
unit-kubernetes-master-1: 12:57:11 INFO unit.kubernetes-master/1.juju-log Checking if snap.kube-proxy.daemon is active (0 / 6)
unit-kubernetes-master-1: 12:57:11 INFO unit.kubernetes-master/1.juju-log Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'get', 'po', '-n', 'kube-system', '-o', 'json', '--request-timeout', '10s']
unit-kubernetes-master-1: 12:57:11 INFO unit.kubernetes-master/1.juju-log Checking system pods status: coredns-7bb4d77796-kw4x8=Running, csi-cinder-controllerplugin-0=Running, csi-cinder-nodeplugin-cvdpl=Running, csi-cinder-nodeplugin-mm62q=Running, csi-cinder-nodeplugin-rshl6=Running, kube-state-metrics-6f586bb967-mkrjr=Running, metrics-server-v0.3.6-f6cf867b4-xnbn8=Running, openstack-cloud-controller-manager-tgwjx=Running, openstack-cloud-controller-manager-z56b8=Running, openstack-cloud-controller-manager-zm4c4=Running
unit-kubernetes-master-1: 12:57:11 INFO unit.kubernetes-master/1.juju-log status-set: active: Kubernetes master running.
unit-kubernetes-master-1: 12:57:12 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-kubernetes-master-1: 12:57:13 INFO unit.kubernetes-master/1.juju-log openstack:28: Reactive main running for hook openstack-relation-departed
unit-kubernetes-master-1: 12:57:28 INFO unit.kubernetes-master/1.juju-log openstack:28: Initializing Apt Layer
unit-kubernetes-master-1: 12:57:28 INFO unit.kubernetes-master/1.juju-log openstack:28: Initializing Snap Layer
unit-kubernetes-master-1: 12:57:29 INFO unit.kubernetes-master/1.juju-log openstack:28: Coordinator: Initializing coordinator layer
unit-kubernetes-master-1: 12:57:30 INFO unit.kubernetes-master/1.juju-log openstack:28: Initializing Leadership Layer (is leader)
unit-kubernetes-master-1: 12:57:30 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-kubernetes-master-1: 12:57:30 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:442:check_cohort_updates
unit-kubernetes-master-1: 12:57:38 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:452:safely_join_cohort
unit-kubernetes-master-1: 12:57:38 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:699:set_app_version
unit-kubernetes-master-1: 12:57:38 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: ../.venv/lib/python3.8/site-packages/charmhelpers/core/host.py:722:wrapped_f
unit-kubernetes-master-1: 12:57:38 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:1189:etcd_data_change
unit-kubernetes-master-1: 12:57:38 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:1220:send_cluster_dns_detail
unit-kubernetes-master-1: 12:57:39 INFO unit.kubernetes-master/1.juju-log openstack:28: Executing ['kubectl', '--kubeconfig=/root/.kube/config', 'get', 'service', '--namespace', 'kube-system', 'kube-dns', '--output', 'json']
unit-kubernetes-master-1: 12:57:39 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:1303:push_service_data
unit-kubernetes-master-1: 12:57:39 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:1320:send_data
unit-kubernetes-master-1: 12:57:39 INFO unit.kubernetes-master/1.juju-log openstack:28: Invoking reactive handler: reactive/kubernetes_master.py:1394:configure_cdk_addons
unit-kubernetes-master-1: 12:57:40 ERROR unit.kubernetes-master/1.juju-log openstack:28: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/reactive/kubernetes_master.py", line 1508, in configure_cdk_addons
generate_openstack_cloud_config().encode('utf-8')
File "lib/charms/layer/kubernetes_common.py", line 500, in generate_openstack_cloud_config
'auth-url = {}'.format(openstack.auth_url),
File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/hooks/relations/openstack-integration/requires.py", line 129, in auth_url
return self._received['auth_url']
File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/hooks/relations/openstack-integration/requires.py", line 63, in _received
return self.relations[0].joined_units.received
File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/endpoints.py", line 582, in __getitem__
return super().__getitem__(self._translate_key(key))
IndexError: list index out of range
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed Traceback (most recent call last):
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/hooks/openstack-relation-departed", line 22, in <module>
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed main()
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed bus.dispatch(restricted=restricted_mode)
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed _invoke(other_handlers)
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed handler.invoke()
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed self._action(*args)
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/reactive/kubernetes_master.py", line 1508, in configure_cdk_addons
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed generate_openstack_cloud_config().encode('utf-8')
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "lib/charms/layer/kubernetes_common.py", line 500, in generate_openstack_cloud_config
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed 'auth-url = {}'.format(openstack.auth_url),
unit-kubernetes-master-1: 12:57:40 WARNING unit.kubernetes-master/1.openstack-relation-departed File "/var/lib/juju/agents/unit-kubernetes-master-1/charm/hooks/relations/openstack-integration/requires.py", line 129, in auth_url
This issue was found in relation to the openstack-