Wrong cloud address used in cross model secret on k8s

[Ian Booth]

On k8s, when a secret is to be read by a consuming unit, an api call is made to the model hosting the owner of the secret to get the connection details for the secret backend. Where the backend is k8s and is the cluster on which the controller was bootstrapped, juju is incorrectly *always* handing out the in cluster address for the kube api server. This breaks the cross model scenario where the consuming model might be on a different cluster.

[Paulo Machado (paulomachado)]

Hi Ian,

the milestone is set to 3.3.2, but will this trickle down to 3.1.x too?

[Ian Booth (wallyworld)]

We aren't planning on any more scheduled 3.1 releases since 3.1 is coming to end of life in April and is currently in security fix only mode. We can look at landing the fix in 3.1 since there might be a final 3.1.8 release and if that does happen, the fix would be included.

[Paulo Machado (paulomachado)]

ACK, thanks

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/16827

[Marcelo Henrique Neppel (neppel)]

Hi Ian! Thanks a lot for the fix. I tested it locally on microk8s and also on AKS+GKE. It's working fine.

It's not working only in one specific situation: sharing secrets between applications in two different controllers residing in the same microk8s installation. Is that intentional?

Steps to reproduce (docker.io/neppel contains the jujud-operator image built from the 7067ce42c9be35e0c5e7d8898854bbd210c64ce1 commit, which was the latest commit in the 3.1 branch when I started to test - I used that to test on both microk8s and AKS+GKE):

sudo snap install juju --channel 3.1/edge

# Owner
git clone https://github.com/PietroPasotti/secrets-demo-charms
cd secrets-demo-charms/owner
charmcraft pack
juju bootstrap microk8s micro1 --config caas-image-repo=docker.io/neppel
juju add-model owner
juju deploy ./*.charm
juju offer owner:secret_id

# Holder
cd ../holder
charmcraft pack
juju bootstrap microk8s micro2 --config caas-image-repo=docker.io/neppel
juju add-model holder
juju deploy ./*.charm
juju consume micro1:admin/owner.owner
juju relate owner holder:secret_id

Error:
unit-holder-0: 19:07:32 ERROR unit.holder/0.juju-log secret_id:0: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 2564, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-holder-0/secret-get', 'secret://0331e832-90c5-47d3-8575-f8c1f8d7029f/cmq05htfvg8s7462f7v0', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 282, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 2920, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 2570, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR getting cluster client: unable to determine legacy status for namespace "owner": Get "https://127.0.0.1:16443/api/v1/namespaces/owner": dial tcp 127.0.0.1:16443: connect: connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 2564, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-holder-0/secret-info-get', 'secret://0331e832-90c5-47d3-8575-f8c1f8d7029f/cmq05htfvg8s7462f7v0', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-holder-0/charm/venv/ops/model.py", line 2930, in _run_for_secret
    re...

[Pedro Guimarães (pguimaraes)]

Hi @neppel, can you test it in the following scenario:

1x juju controller in a VM or even LXC
Add the k8s cluster(s) to the controller
2x models in k8s, connected via CMR.

This is the standard Field deployment. Juju controllers are generally deployed outside of k8s because juju has HA only in VM, but not with juju-k8s.