metallb-controller errors during installation

Bug #1988410 reported by Konstantinos Kaskavelis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MetalLB Operator
Fix Released
Critical
Stone Preston

Bug Description

Solutions QA team has a failing run where metallb-controller fails to get installed with the following message: hook failed: "install"

metallb layer is built on top of the kubernetes layer, using a juju controller, which in turn is deployed on top of MAAS.

From the logs we see:

7c9fe6a6-cd8c-427a-8f54-60b555f9e7c0: application-metallb-controller 2022-08-31 21:35:32 INFO unit.metallb-controller/0.juju-log server.go:316 Creating pod security policy with K8s API
7c9fe6a6-cd8c-427a-8f54-60b555f9e7c0: application-metallb-controller 2022-08-31 21:35:32 ERROR unit.metallb-controller/0.juju-log server.go:316 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 192, in <module>
    main(MetalLBControllerCharm)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/ops/main.py", line 400, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/ops/main.py", line 138, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/ops/framework.py", line 216, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/ops/framework.py", line 628, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/ops/framework.py", line 673, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 71, in _on_start
    utils.create_k8s_objects(self._stored.namespace)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/src/utils.py", line 16, in create_k8s_objects
    create_pod_security_policy_with_api(namespace=namespace)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/src/utils.py", line 77, in create_pod_security_policy_with_api
    api_instance.create_pod_security_policy(body, pretty=True)
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/api/policy_v1beta1_api.py", line 173, in create_pod_security_policy
    (data) = self.create_pod_security_policy_with_http_info(body, **kwargs) # noqa: E501
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/api/policy_v1beta1_api.py", line 243, in create_pod_security_policy_with_http_info
    return self.api_client.call_api(
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/api_client.py", line 340, in call_api
    return self.__call_api(resource_path, method,
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/api_client.py", line 172, in __call_api
    response_data = self.request(
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/api_client.py", line 382, in request
    return self.rest_client.POST(url,
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/rest.py", line 272, in POST
    return self.request("POST", url,
  File "/var/lib/juju/agents/unit-metallb-controller-0/charm/venv/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '2c147e60-40e6-4f59-bb5c-1db3851823bf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'fb4c65ab-9779-4517-aa0d-9df4f69f8b25', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e433f2b8-7dbd-4ce9-94ee-ed14e6396177', 'Date': 'Wed, 31 Aug 2022 21:35:32 GMT', 'Content-Length': '206'})
HTTP response body: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "the server could not find the requested resource",
  "reason": "NotFound",
  "details": {},
  "code": 404
}

22566f82-0511-40b6-85d7-f7dbcd8583bc: machine-1 2022-08-31 21:35:32 DEBUG juju.apiserver request_notifier.go:189 <- [174] unit-telegraf-0 {"request-id":709,"type":"LeadershipService","version":2,"request":"ClaimLeadership","params":"'params redacted'"}
22566f82-0511-40b6-85d7-f7dbcd8583bc: machine-1 2022-08-31 21:35:32 DEBUG juju.apiserver request_notifier.go:221 -> [174] unit-telegraf-0 3.347768ms {"request-id":709,"response":"'body redacted'"} LeadershipService[""].ClaimLeadership
7c9fe6a6-cd8c-427a-8f54-60b555f9e7c0: application-metallb-controller 2022-08-31 21:35:32 ERROR juju.worker.caasoperator.uniter.metallb-controller/0.operation runhook.go:140 hook "install" (via hook dispatching script: dispatch) failed: exit status 1

Test run:

https://solutions.qa.canonical.com/testruns/testRun/36787ef2-8972-44e1-99e3-df8fc693e833

Log artifacts can be found in the link of the bottom of the url above.

tags: added: cdo-qa cdo-release-blocker foundations-engine
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :
Changed in operator-metallb:
status: New → Confirmed
status: Confirmed → In Progress
assignee: nobody → Stone Preston (stonepreston)
milestone: none → 1.25+ck1
tags: added: backport-needed review-needed
Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

Bumping to critical as this affects metal deployment.

Changed in operator-metallb:
importance: Undecided → Critical
Adam Dyess (addyess)
Changed in operator-metallb:
status: In Progress → Fix Committed
Revision history for this message
Adam Dyess (addyess) wrote (last edit ):

Testing is available on these charms from charm channels 1.25/edge

juju deploy metallb-speaker --channel=1.25/edge
juju deploy metallb-controller --channel=1.25/edge

Adam Dyess (addyess)
tags: removed: backport-needed
Adam Dyess (addyess)
Changed in operator-metallb:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.