no-proxy CIDRs not handled by dependencies

[Peter Jose De Sousa]

Hello,

When deploying the vsphere cloud provider inside of environments with proxies, setting the juju model-config no-proxy environment variables with CIDRs will not work.

This is because the underlaying dependecies will make HTTPs requests to the kubernetes-control-plane, which will ultimately be proxied.

[Logs]

➜ esa juju debug-log -i vsphere-cloud-provider/0
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying cloud-provider-vsphere version: v1.24
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider secret data for server 10.246.152.100
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider ConfigMap Data for vcenter Boston
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Replacing Image: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.24.0 with rocks.canonical.com:443/cdk/cloud-provider-vsphere/cpi/release/manager:v1.24.0
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider Control Node Selector as node-role.kubernetes.io/control-plane: ""
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Adding provider tolerations from control-plane
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying ServiceAccount/kube-system/cloud-controller-manager
unit-vsphere-cloud-provider-0: 10:33:49 ERROR unit.vsphere-cloud-provider/0.juju-log kube-control:17: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/http_proxy.py", line 267, in handle_request
    raise ProxyError(msg)
httpcore.ProxyError: 403 Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 216, in <module>
    main(VsphereCloudProviderCharm)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event) # noqa
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 196, in _merge_config
    self._install_or_upgrade()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 204, in _install_or_upgrade
    controller.apply_manifests()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 232, in apply_manifests
    self.apply_resources(*self.resources)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 246, in apply_resources
    self.client.apply(rsc.resource, force=True)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 424, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 293, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 244, in request
    resp = self.send(req)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 216, in send
    return self._client.send(req, stream=stream)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 908, in send
    response = self._send_handling_auth(
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 936, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 973, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 1009, in _send_single_request
    response = transport.handle_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ProxyError: 403 Forbidden
unit-vsphere-cloud-provider-0: 10:33:50 ERROR juju.worker.uniter.operation hook "kube-control-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-vsphere-cloud-provider-0: 10:33:50 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook

[Workaround]

Set no proxy to include IP or FQDN of the Kubernetes Control Plane (LB/IP) (Not a CIDR)

Thanks,
Peter

[Peter Jose De Sousa (pjds)]

Updated workaround.

[Adam Dyess (addyess)]

I believe this could affect multiple charms using a particular library:

Addressing the situation here as i understand it
https://github.com/canonical/ops-lib-manifest/pull/13

[Adam Dyess (addyess)]

upon further evaluation, adding the api loadbalancer ip to the list of hostnames in `juju model-config no-proxy` is probably the best work-around.

There aren't many python http client libraries that support cidr ranges, but only support a list of hostnames

In this SPECIFIC case, the client library used by the charm is httpx

Docs: https://www.python-httpx.org/environment_variables/#no_proxy

> Valid values: a comma-separated list of hostnames/urls

Notice that CIDR networks aren't supported here. Expanding the CIDR range here doesn't seem like a good idea, so I think that work-around is the best one to leave in place -- maybe even close the bug as a "Won't fix"

Reversion Patch: https://github.com/canonical/ops-lib-manifest/pull/14

[Kevin W Monroe (kwmonroe)]

Per comment 4, changing the underlying lib that causes all these projects to be affected won't solve the issue. Given the workaround (explicit fqdn/ip in the model-config no-proxy), I'm closing these as won't fix.

We'll do a better job of calling out this scenario in our proxy docs to help future travelers debug their proxied envs:

https://github.com/charmed-kubernetes/kubernetes-docs/issues/749