many -mysql units failed on zip: not a valid zip file

[Marian Gasparovic]

During sunbeam bootstrap many -mysql units showed crash loop backoff: back-off 5m0s restarting failed container=charm...

Logs of the pods show

2024-04-22T14:13:07.537Z [container-agent] 2024-04-22 14:13:07 INFO juju.worker.uniter uniter.go:363 unit "horizon-mysql/0" started
2024-04-22T14:13:07.538Z [container-agent] 2024-04-22 14:13:07 INFO juju.worker.uniter uniter.go:689 resuming charm install
2024-04-22T14:13:07.538Z [container-agent] 2024-04-22 14:13:07 INFO juju.worker.uniter uniter.go:347 unit "horizon-mysql/0" shutting down: preparing operation "install ch:amd64/jammy/mysql-k8s-127" for horizon-mysql/0: zip: not a valid zip file
2024-04-22T14:13:17.440Z [pebble] Check "readiness" failure 1 (threshold 3): received non-20x status code 404
2024-04-22T14:13:17.440Z [pebble] Check "liveness" failure 1 (threshold 3): received non-20x status code 404
2024-04-22T14:13:27.439Z [pebble] Check "liveness" failure 2 (threshold 3): received non-20x status code 404
2024-04-22T14:13:27.439Z [pebble] Check "readiness" failure 2 (threshold 3): received non-20x status code 404
2024-04-22T14:13:37.436Z [pebble] Check "liveness" failure 3 (threshold 3): received non-20x status code 404
2024-04-22T14:13:37.436Z [pebble] Check "liveness" failure threshold 3 hit, triggering action
2024-04-22T14:13:37.436Z [pebble] Check "readiness" failure 3 (threshold 3): received non-20x status code 404
2024-04-22T14:13:37.436Z [pebble] Check "readiness" failure threshold 3 hit, triggering action
2024-04-22T14:13:47.437Z [pebble] Check "liveness" failure 4 (threshold 3): received non-20x status code 404
2024-04-22T14:13:47.437Z [pebble] Check "readiness" failure 4 (threshold 3): received non-20x status code 404
2024-04-22T14:13:49.074Z [pebble] Exiting on terminated signal.
2024-04-22T14:13:49.079Z [pebble] Stopping all running services.
2024-04-22T14:13:57.436Z [pebble] Check "liveness" failure 5 (threshold 3): received non-20x status code 404
2024-04-22T14:13:57.436Z [pebble] Check "readiness" failure 5 (threshold 3): received non-20x status code 404
2024-04-22T14:14:07.437Z [pebble] Check "liveness" failure 6 (threshold 3): received non-20x status code 404
2024-04-22T14:14:07.437Z [pebble] Check "readiness" failure 6 (threshold 3): received non-20x status code 404
2024-04-22T14:14:17.437Z [pebble] Check "liveness" failure 7 (threshold 3): received non-20x status code 404
2024-04-22T14:14:17.437Z [pebble] Check "readiness" failure 7 (threshold 3): received non-20x status code 404

Logs - https://oil-jenkins.canonical.com/artifacts/7c8529af-cfcc-4405-81c7-2b778832ccbb/index.html

[Guillaume Boutry (gboutry)]

This looks like an issue downloading the charm between the container agent and the juju controller. Are there proxy involved ?

[Marian Gasparovic (marosg)]

No proxy, proxy_required: false in manifest file