Enabling maas tls requires restart of juju controllers

[Nishant Dash]

If juju is setup bootstrapped to a maas cloud without tls and then tls is enabled in maas later on, I would expect that simply running `juju update-cloud ...` will be. However it turns out that I need to restart the juju controllers as well to get it working.

Here are the steps I took:
1. bootstrap maas cloud to juju
```
cat clouds.yaml
clouds:
  maas_cloud:
    type: maas
    auth-types: [oauth1]
    endpoint: http://<maas-vip>:80/MAAS
```

2. enable tls after the fact with maas following [1], which at this time is
```
maas config-tls enable /var/snap/maas/common/x.key /var/snap/maas/common/x.crt
maas login $PROFILE https://<maas-fqdn-or-vip-depending-on-your-certs>:5443/MAAS ...
```

NOTE ^ I need to maas re-login otherwise certain commands like maas machine read work but `maas subnets read` does not work.

3. write a new cloud yaml with the endpoint updated and the cert chain added
```
cat << EOF > maas-cloud-tls.yaml
maas_cloud:
  type: maas
  auth-types: [oauth1]
  endpoint: https://<maas-fqdn-or-vip-depending-on-your-certs>:5443/MAAS
  ca-certificates:
  - |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
    ...
EOF
```

4. update cloud
```
juju update-cloud maas_cloud -f maas-cloud-tls.yaml
# accept for both client and controller
```

5. run juju spaces (to mimic maas subnets read)
```
juju spaces
ERROR cannot list spaces: getting environ: creating environ for model "maas-infra" (1e82964a-e3ae-4b95-8142-01d8b7b019dc): creating MAAS environ: getting MAAS controller: Get "https://<maas-fqdn-or-vip-depending-on-your-certs>:5443/MAAS/api/2.0/version/": tls: failed to verify certificate: x509: certificate signed by unknown authority
```
NOTE that this ca cert chain has been updated on the controller vm themselves so the vms trust it

6. rolling restart `jujud-machine-*.service` on all juju controllers

7. juju spaces works then, and I even tested further with a juju deploy,etc...

Here are the relevant versions:
- juju 2.9.44 (server + client)
- maas 3.2/stable (3.2.9-12055-g.c3d5597a7)

[1] https://maas.cloud.cbh.kth.se/MAAS/docs/ui/how-to-enable-maas-native-tls.html

[Nishant Dash (dash3)]

Speaking to Joseph, they mentioned its possibly relating to transport in the MAAS client (gomaasapi) itself

[Anton Troyanov (troyanov)]

To what I know, in Go root certificates are loaded only once when the application is started.
Thats why rolling restart helps (app will pick up newly certificates from the trust root)

https://github.com/golang/go/issues/41888

[Joseph Phillips (manadart)]

Triaged as RFE, but it looks like our hands are tied until the Go limitations are addressed.