In test run https://solutions.qa.canonical.com/testruns/d3584e68-704a-4951-b98b-17e56b48cbf6/, the bootstrapping of a juju controller on a snapped microk8s fails with the following message:
===========
2023-09-06-01:59:31 root DEBUG [localhost]: juju add-k8s snapped-microk8s_cloud --client --storage microk8s-hostpath
2023-09-06-01:59:34 fce.juju_snapped-microk8s_controller INFO Finished step: juju_snapped-microk8s_controller:snapped_microk8s_setup_controller. Time elapsed (hh:mm:ss.ms): 0:00:02.677265
2023-09-06-01:59:34 fce.juju_snapped-microk8s_controller INFO Starting step: juju_snapped-microk8s_controller:bootstrap
2023-09-06-01:59:34 root DEBUG ['juju', 'bootstrap', '--config', 'controller-service-type=loadbalancer', '--bootstrap-constraints', 'arch=amd64 ', '--config', 'bootstrap-timeout=1800', '--model-default', '/home/ubuntu/project/generated/juju_snapped-microk8s_controller/model_defaults.yaml', 'snapped-microk8s_cloud', 'popocatepetl']
2023-09-06-01:59:34 root DEBUG [localhost]: juju bootstrap --config controller-service-type=loadbalancer --bootstrap-constraints 'arch=amd64 ' --config bootstrap-timeout=1800 --model-default /home/ubuntu/project/generated/juju_snapped-microk8s_controller/model_defaults.yaml snapped-microk8s_cloud popocatepetl
2023-09-06-01:59:34 root DEBUG Creating Juju controller "popocatepetl" on snapped-microk8s_cloud/localhost
2023-09-06-01:59:34 root DEBUG Bootstrap to Kubernetes cluster identified as microk8s/localhost
2023-09-06-01:59:35 root DEBUG Creating k8s resources for controller "controller-popocatepetl"
2023-09-06-02:00:04 root DEBUG Downloading images
2023-09-06-02:01:47 root DEBUG Starting controller pod
2023-09-06-02:01:47 root DEBUG Bootstrap agent now started
2023-09-06-02:01:47 root DEBUG Contacting Juju controller at 10.246.64.200 to verify accessibility...
2023-09-06-02:01:51 root DEBUG ERROR unable to contact api server after 0 attempts: unable to connect to API: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "juju-ca")
2023-09-06-02:07:46 root ERROR [localhost] Command failed: juju bootstrap --config controller-service-type=loadbalancer --bootstrap-constraints 'arch=amd64 ' --config bootstrap-timeout=1800 --model-default /home/ubuntu/project/generated/juju_snapped-microk8s_controller/model_defaults.yaml snapped-microk8s_cloud popocatepetl
===========
The juju controller can't be accessed because the certificate can't be verified. Did something around the certificates change in juju 3.2.3?
More logs and configs can be found here: https://oil-jenkins.canonical.com/artifacts/d3584e68-704a-4951-b98b-17e56b48cbf6/index.html
We are not hitting this issue consistently with 3.2.3, there are multiple runs that pass the microk8s bootstrapping.