Juju client periodically loses proxy config for k8s controller

[Kian Parvin]

Hi,

I have noticed an oddity on a juju client. This client is running on a Bastion host and is connected to 2 controllers, one is a machine controller and another is a k8s controller. Connecting and using both controllers was working fine until a few weeks ago when connecting to the k8s controller stopped working and it was found that within the controllers.yaml file, the proxy-config section (for the k8s controller) had gone missing. I had chalked this up to someone manually editing the file and fortunately we had a backup of the original file (specifically because we were trying to register users to k8s controller unsuccessfully due to this bug - https://bugs.launchpad.net/juju/+bug/1996120). Nonetheless, manually adding the proxy-config back into the file resolved the issue. Unfortunately this has happened again, where the proxy-config has gone missing. Some details are below:

Original Controllers.yaml file (only the section for the k8s controller):

controllers:
    comsys-ps5-staging:
        uuid: d381ade1-c83f-412f-8793-4a4081e81a21
        api-endpoints: ['<Removed>']
        dns-cache: {localhost: [127.0.0.1]}
        ca-cert: |
            <Removed>
        cloud: ps5-staging
        type: kubernetes
        agent-version: 2.9.34
        controller-machine-count: 0
        active-controller-machine-count: 0
        machine-count: 0
        proxy-config:
            config:
                api-host: <Removed>
                ca-cert: |
                    <Removed>
                namespace: controller-comsys-ps5-staging
                remote-port: "17070"
                service: controller-service
                service-account-token: <Removed>
            type: kubernetes-port-forward
current-controller: comsys-ps5-staging

Logs from today to illustrate the problem:

stg-canonical-livepatch@comsys-bastion-ps5:~$ date
Mon Dec 5 08:39:54 UTC 2022

stg-canonical-livepatch@comsys-bastion-ps5:~$ cat .local/share/juju/controllers.yaml
controllers:
  comsys-ps5-staging:
    uuid: d381ade1-c83f-412f-8793-4a4081e81a21
    api-endpoints: ['<Removed>']
    dns-cache: {localhost: [127.0.0.1]}
    ca-cert: |
      <Removed>
    cloud: ps5-staging
    type: kubernetes
    agent-version: 2.9.34
    controller-machine-count: 0
    active-controller-machine-count: 0
    machine-count: 0
  prodstack-is-beta:
    uuid: ccf3a6e2-8f8e-4c4b-8caa-e425a9657be1
    api-endpoints: ['<Removed>']
    ca-cert: |
      <Removed>
    cloud: ""
    agent-version: 2.9.32
    controller-machine-count: 0
    active-controller-machine-count: 0
current-controller: prodstack-is-beta

stg-canonical-livepatch@comsys-bastion-ps5:~$ juju version
2.9.37-ubuntu-amd64

In Summary:
- The proxy-config option in controllers.yaml is periodically being deleted.
- I initially thought the deletion was due to a user editing the file, but it has occurred twice now.
- Unfortunately I'm not sure how to replicate this but I'll keep a close eye on it and try to see if I can trigger the behavior.
- And as a final aside, this could still be user error, perhaps there's a command or some flag that is causing this proxy-config to be removed?