Impossible to set juju-http(s)-proxy in model-default with Juju 3.X in proxified environment

[DUFOUR Olivier]

This is partially related to LP#2038416, where for some reason it is not possible anymore to deploy Juju 3.X controllers with juju-http(s)-proxy environment variables, and legacy variables such as http-proxy and https-proxy are to be used ...

There is however a supplementary catch because of this new behaviour:
After bootstrapping Juju 3.X controllers with http-proxy and https-proxy as Proxy environment, it is impossible to set in the model-default settings juju-http-proxy or juju-https-proxy even if http-proxy/https-proxy/no-proxy is empty. (See attached file for indepth reproducing tests)
This means for every new model added to the controller, it gets necessary to redefine the proxy variables manually instead of being able to use model-default variables.

Tested environment :
* Ubuntu 22.04
* MAAS : 3.3
* Juju : 3.1.6 and 3.3.0

The reproducing steps are :
* Bootstrap a controller with http-proxy and https-proxy
* reset the variables http-proxy, https-proxy and no-proxy
* try to set the variables juju-http-proxy, juju-https-proxy
 --> get greeted with the error : ERROR cannot specify both legacy proxy values and juju proxy values

What is working :
* setting any another value, aside from juju-http(s)-proxy, in model-default seems to work
* it is possible in model-config of any other model to set the juju-http-proxy and juju-https-proxy

What doesn't work :
* setting the juju-http-proxy and juju-https-proxy in model-default after bootstrapping the controller
* setting the juju-http-proxy and juju-https-proxy in model-default after creating a new model
* setting the juju-http-proxy and juju-https-proxy in model-default after resetting all proxy variables in model-default (including APT and snaps)
* only juju-http-proxy and juju-https-proxy are failing in model-default

[Ian Booth (wallyworld)]

See also https://bugs.launchpad.net/juju/+bug/2038416

Quick reply without digging in deeply....

It's complicated because there's several components that need to use proxy config:
- juju cli client
- juju agents (incl controller, machine agent, unit agent)
- pebble
- charms (to configure workloads / tooling eg curl)

As mentioned elsewhere, the "juju-" variants were introduced to deal with the no-proxy cidr limitations that existed at the time for things like curl. And the approach was done before pebble entered the mix.

If you had asked me, I would have said that the juju components (cli and agents) would have used the "juju-" proxy values, if set. So there should be no need to also have to set the "http-" variants in order to bootstrap. That's why the "cannot set both" rule is in place as far as I recall.

So in your scenario, setting the "juju-" values and seeing

ERROR cannot deploy controller application: deploying charmhub controller charm: resolving "ch:juju-controller": resolving with preferred channel: Post "https://api.charmhub.io/v2/charms/refresh": dial tcp 185.125.188.55:443: connect: network is unreachable

seems to be a bug. Juju should be configuring the http client used to access charmhub based off those "juju-" values. If that gets fixed, you don't need to set the "http-" values, just the "juju-" ones.

[Nobuto Murata (nobuto)]

> Juju should be configuring the http client used to access charmhub based off those "juju-" values. If that gets fixed, you don't need to set the "http-" values, just the "juju-" ones.

Exactly. Olivier filed this since we don't have a privilege to reopen: https://bugs.launchpad.net/juju/+bug/2038416

As both Yoshi and Olivier are working on the same project, we are fine one to be marked as a duplicate of the other as long as the mentioned behavior above is achieved.

[DUFOUR Olivier (odufourc)]

This is confirmed to be still an ongoing issue with Juju 3.4 release.

I have found a temporary workaround, with the option "--controller-charm-path" although this is not perfectly ideal.
This is for a deployment with Juju 3.4, with having all your proxy settings (with juju-http(s)-proxy) in the default-model configuration as a file in "juju-default-config.yaml"

* download the controller charm for the wanted release of Juju
--> juju download juju-controller --channel 3.4/stable --base ubuntu@22.04

* bootstrap the controller
--> juju bootstrap --bootstrap-base ubuntu@22.04 --controller-charm-path ./juju-controller_r79.charm --model-default ./juju-default-config.yaml maas_cloud maas-juju-3-4

* enable-ha with the controllers
--> juju enable-ha

* After all the controllers are now working HA and "juju controllers --refresh" do show the scale of 3/3
--> juju refresh -m controller controller --switch ch:juju-controller --channel 3.4/stable

After this all models, including the controller model, will have the juju-http(s)-proxy being defined and used

[Muhammad Ahmad (ahmadfsbd)]

Hitting the same issue with juju-3.4 with FCE. Can we speed this up please as this is critical for customer projects and CI deployments.

[Ian Booth (wallyworld)]

Until we can get this fixed, the workaround in comment #4 should work?