charm errors during update-status hook with 502 Gateway Error
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Azure Cloud Provider |
Fix Released
|
Medium
|
Mateo Florido | ||
Charm AWS Kubernetes Storage |
Fix Released
|
Medium
|
Adam Dyess | ||
Charm GCP Kubernetes Storage |
Fix Released
|
Medium
|
Adam Dyess | ||
Multus Charm |
Fix Released
|
Medium
|
Mateo Florido | ||
OPA Gatekeeper Operator |
Fix Released
|
Medium
|
Mateo Florido | ||
SR-IOV CNI Charm |
Fix Released
|
Medium
|
Adam Dyess | ||
vSphere Cloud Provider Charm |
Fix Released
|
Medium
|
Adam Dyess |
Bug Description
During a test of k8s 1.26 GA, we bumped into an error where the aws-k8s-storage charm went into an error state because a request to kubeapi-
The failure happened during the `test_audit_
from the aws-k8s-storage juju log:
unit-aws-
unit-aws-
Traceback (most recent call last):
File "/var/lib/
main(
File "/var/lib/
_emit_
File "/var/lib/
event_
File "/var/lib/
framework.
File "/var/lib/
self.
File "/var/lib/
custom_
File "/var/lib/
unready = self.collector.
File "/var/lib/
return sorted(
File "/var/lib/
for obj in manifest.status()
File "/var/lib/
return frozenset(_ for _ in self.installed_
File "/var/lib/
next_rsc = self.client.get(
File "/var/lib/
return self._client.
File "/var/lib/
return self.handle_
File "/var/lib/
self.
File "/var/lib/
raise transform_
File "/var/lib/
resp.
File "/var/lib/
raise HTTPStatusError
httpx.HTTPStatu
For more information check: https:/
looking forward at what was 18.215.245.193 (172.31.35.42), it was the kubeapi-
2022/12/09 23:20:56 [error] 42357#42357: *439 no live upstreams while connecting to upstream, client: 172.31.40.35, server: server_443, request: "PUT /apis/coordinat
2022/12/09 23:20:56 [error] 42357#42357: *439 no live upstreams while connecting to upstream, client: 172.31.40.35, server: server_443, request: "PUT /apis/coordinat
2022/12/09 23:20:56 [error] 42357#42357: *439 no live upstreams while connecting to upstream, client: 172.31.40.35, server: server_443, request: "PUT /apis/coordinat
172.31.40.35 is one of the k8s-workers (kubernetes-
however one of the k8s-cp units (the leader) was restarting the kube-apiserver.
09 Dec 2022 23:20:35Z juju-unit executing running config-changed hook
09 Dec 2022 23:20:55Z workload maintenance Restarting snap.kube-
09 Dec 2022 23:21:47Z juju-unit idle
I'm not sure what an ideal fix here would be maybe doing an additional few tries with a back off in case services are restarting somewhere?
The testrun can be found at:
https:/
with the crashdump at:
https:/
Changed in charm-aws-k8s-storage: | |
status: | New → In Progress |
summary: |
- charm errors if querying kube-system/secrets/aws-secret times errors + charm errors during update-status hook with 502 Gateway Error |
Changed in opa-gatekeeper-operator: | |
importance: | Undecided → Medium |
assignee: | nobody → Adam Dyess (addyess) |
milestone: | none → 1.26+ck1 |
Changed in charm-sriov-cni: | |
importance: | Undecided → Medium |
assignee: | nobody → Adam Dyess (addyess) |
milestone: | none → 1.26+ck1 |
Changed in charm-vsphere-cloud-provider: | |
status: | New → In Progress |
milestone: | 1.25 → 1.26+ck1 |
Changed in charm-sriov-cni: | |
status: | New → In Progress |
Changed in opa-gatekeeper-operator: | |
assignee: | Adam Dyess (addyess) → Mateo Florido (mateoflorido) |
status: | New → In Progress |
Changed in charm-aws-k8s-storage: | |
status: | In Progress → Fix Committed |
Changed in charm-gcp-k8s-storage: | |
status: | In Progress → Fix Committed |
Changed in charm-sriov-cni: | |
status: | In Progress → Fix Committed |
Changed in charm-vsphere-cloud-provider: | |
status: | In Progress → Fix Committed |
Changed in charm-azure-cloud-provider: | |
status: | In Progress → Fix Committed |
Changed in charm-multus: | |
status: | In Progress → Fix Committed |
Changed in opa-gatekeeper-operator: | |
status: | In Progress → Fix Committed |
tags: | added: backport-needed |
tags: | removed: backport-needed |
Changed in charm-aws-k8s-storage: | |
status: | Fix Committed → Fix Released |
Changed in charm-azure-cloud-provider: | |
status: | Fix Committed → Fix Released |
Changed in charm-gcp-k8s-storage: | |
status: | Fix Committed → Fix Released |
Changed in charm-multus: | |
status: | Fix Committed → Fix Released |
Changed in opa-gatekeeper-operator: | |
status: | Fix Committed → Fix Released |
Changed in charm-sriov-cni: | |
status: | Fix Committed → Fix Released |
Changed in charm-vsphere-cloud-provider: | |
status: | Fix Committed → Fix Released |
Affected library: /github. com/canonical/ ops-lib- manifest/ pull/11
https:/