Unit in error when deploying with an overlay

Bug #1999532 reported by Mateo Florido
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vSphere Cloud Provider Charm
Fix Released
Medium
Mateo Florido

Bug Description

When trying to deploy the vSphere-Cloud-Provider charm using the following overlay:
```
description: Charmed Kubernetes overlay to add native vSphere support.
applications:
  kubernetes-control-plane:
    options:
      allow-privileged: "true"
  vsphere-integrator:
    charm: vsphere-integrator
    num_units: 1
    trust: true
  vsphere-cloud-provider:
    charm: vsphere-cloud-provider
relations:
- - vsphere-cloud-provider:certificates
  - easyrsa:client
- - vsphere-cloud-provider:kube-control
  - kubernetes-control-plane:kube-control
- - vsphere-cloud-provider:external-cloud-provider
  - kubernetes-control-plane:external-cloud-provider
- - vsphere-cloud-provider:vsphere-integration
  - vsphere-integrator:clients
```

The charm enters in error state because the kube-apiserver is not yet ready to receive requests. Running `juju resolve vsphere-cloud-provider/0` after a while fixes the problem.

juju debug-log:

```
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Applying cloud-provider-vsphere version: v1.24
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Applying provider secret data for server 10.246.152.100
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Applying provider ConfigMap Data for vcenter Boston
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Replacing Image: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.24.0 with rocks.canonical.com:443/cdk/cloud-provider-vsphere/cpi/release/manager:v1.24.0
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Applying provider Control Node Selector as node-role.kubernetes.io/control-plane: ""
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Adding provider tolerations from control-plane
unit-vsphere-cloud-provider-0: 16:03:58 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:18: Applying ServiceAccount/kube-system/cloud-controller-manager
unit-vsphere-cloud-provider-0: 16:03:59 DEBUG unit.vsphere-cloud-provider/0.juju-log kube-control:18: HTTP Request: PATCH https://10.246.154.107:6443/api/v1/namespaces/kube-system/serviceaccounts/cloud-controller-manager?force=true&fieldManager=vsphere-cloud-provider-cloud-provider-vsphere "HTTP/1.1 401 Unauthorized"
unit-vsphere-cloud-provider-0: 16:03:59 ERROR unit.vsphere-cloud-provider/0.juju-log kube-control:18: Failed Applying ServiceAccount/kube-system/cloud-controller-manager
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://10.246.154.107:6443/api/v1/namespaces/kube-system/serviceaccounts/cloud-controller-manager?force=true&fieldManager=vsphere-cloud-provider-cloud-provider-vsphere'
For more information check: https://httpstatuses.com/401

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 246, in apply_resources
    self.client.apply(rsc.resource, force=True)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 424, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 293, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: Unauthorized
unit-vsphere-cloud-provider-0: 16:03:59 ERROR unit.vsphere-cloud-provider/0.juju-log kube-control:18: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://10.246.154.107:6443/api/v1/namespaces/kube-system/serviceaccounts/cloud-controller-manager?force=true&fieldManager=vsphere-cloud-provider-cloud-provider-vsphere'
For more information check: https://httpstatuses.com/401

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 216, in <module>
    main(VsphereCloudProviderCharm)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 196, in _merge_config
    self._install_or_upgrade()
  File "./src/charm.py", line 204, in _install_or_upgrade
    controller.apply_manifests()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 232, in apply_manifests
    self.apply_resources(*self.resources)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 246, in apply_resources
    self.client.apply(rsc.resource, force=True)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 424, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 293, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: Unauthorized
unit-vsphere-cloud-provider-0: 16:03:59 ERROR juju.worker.uniter.operation hook "kube-control-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
```

Changed in charm-vsphere-cloud-provider:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Mateo Florido (mateoflorido)
Revision history for this message
Mateo Florido (mateoflorido) wrote :
Changed in charm-vsphere-cloud-provider:
status: In Progress → Fix Committed
Adam Dyess (addyess)
Changed in charm-vsphere-cloud-provider:
milestone: none → 1.26+ck1
Adam Dyess (addyess)
tags: added: backport-needed
Adam Dyess (addyess)
tags: removed: backport-needed
Changed in charm-vsphere-cloud-provider:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.