no-proxy CIDRs not handled by dependencies

Bug #2002646 reported by Peter Jose De Sousa
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Azure Cloud Provider
Won't Fix
Medium
Adam Dyess
Charm AWS Kubernetes Storage
Won't Fix
Medium
Adam Dyess
Charm GCP Kubernetes Storage
Won't Fix
Medium
Adam Dyess
KubeVirt Charm
Won't Fix
Medium
Adam Dyess
Multus Charm
Won't Fix
Medium
Adam Dyess
OPA Gatekeeper Operator
Won't Fix
Medium
Adam Dyess
SR-IOV CNI Charm
Won't Fix
Medium
Adam Dyess
SR-IOV Network Device Plugin Charm
Won't Fix
Medium
Adam Dyess
vSphere Cloud Provider Charm
Won't Fix
Medium
Adam Dyess

Bug Description

Hello,

When deploying the vsphere cloud provider inside of environments with proxies, setting the juju model-config no-proxy environment variables with CIDRs will not work.

This is because the underlaying dependecies will make HTTPs requests to the kubernetes-control-plane, which will ultimately be proxied.

[Logs]

➜ esa juju debug-log -i vsphere-cloud-provider/0
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying cloud-provider-vsphere version: v1.24
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider secret data for server 10.246.152.100
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider ConfigMap Data for vcenter Boston
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Replacing Image: gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.24.0 with rocks.canonical.com:443/cdk/cloud-provider-vsphere/cpi/release/manager:v1.24.0
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying provider Control Node Selector as node-role.kubernetes.io/control-plane: ""
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Adding provider tolerations from control-plane
unit-vsphere-cloud-provider-0: 10:33:49 INFO unit.vsphere-cloud-provider/0.juju-log kube-control:17: Applying ServiceAccount/kube-system/cloud-controller-manager
unit-vsphere-cloud-provider-0: 10:33:49 ERROR unit.vsphere-cloud-provider/0.juju-log kube-control:17: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/connection_pool.py", line 253, in handle_request
    raise exc
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpcore/_sync/http_proxy.py", line 267, in handle_request
    raise ProxyError(msg)
httpcore.ProxyError: 403 Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 216, in <module>
    main(VsphereCloudProviderCharm)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 438, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event) # noqa
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 856, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/framework.py", line 931, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 196, in _merge_config
    self._install_or_upgrade()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/./src/charm.py", line 204, in _install_or_upgrade
    controller.apply_manifests()
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 232, in apply_manifests
    self.apply_resources(*self.resources)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/ops/manifests/manifest.py", line 246, in apply_resources
    self.client.apply(rsc.resource, force=True)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 424, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/client.py", line 293, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 244, in request
    resp = self.send(req)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/lightkube/core/generic_client.py", line 216, in send
    return self._client.send(req, stream=stream)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 908, in send
    response = self._send_handling_auth(
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 936, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 973, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_client.py", line 1009, in _send_single_request
    response = transport.handle_request(request)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/var/lib/juju/agents/unit-vsphere-cloud-provider-0/charm/venv/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ProxyError: 403 Forbidden
unit-vsphere-cloud-provider-0: 10:33:50 ERROR juju.worker.uniter.operation hook "kube-control-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-vsphere-cloud-provider-0: 10:33:50 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook

[Workaround]

Set no proxy to include IP or FQDN of the Kubernetes Control Plane (LB/IP) (Not a CIDR)

Thanks,
Peter

description: updated
Revision history for this message
Peter Jose De Sousa (pjds) wrote :

Updated workaround.

description: updated
George Kraft (cynerva)
Changed in charm-vsphere-cloud-provider:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Adam Dyess (addyess) wrote :

I believe this could affect multiple charms using a particular library:

Addressing the situation here as i understand it
https://github.com/canonical/ops-lib-manifest/pull/13

Changed in charm-vsphere-cloud-provider:
milestone: none → 1.26+ck2
Changed in charm-aws-k8s-storage:
importance: Undecided → Medium
Changed in charm-azure-cloud-provider:
importance: Undecided → Medium
Changed in charm-gcp-k8s-storage:
importance: Undecided → Medium
Changed in charm-kubevirt:
importance: Undecided → Medium
Changed in charm-multus:
importance: Undecided → Medium
Changed in opa-gatekeeper-operator:
importance: Undecided → Medium
Changed in charm-sriov-cni:
importance: Undecided → Medium
Changed in charm-sriov-network-device-plugin:
importance: Undecided → Medium
status: New → In Progress
Changed in charm-vsphere-cloud-provider:
status: Triaged → In Progress
Changed in charm-sriov-cni:
status: New → In Progress
Changed in opa-gatekeeper-operator:
status: New → In Progress
Changed in charm-multus:
status: New → In Progress
Changed in charm-kubevirt:
status: New → In Progress
Changed in charm-gcp-k8s-storage:
status: New → In Progress
Changed in charm-azure-cloud-provider:
status: New → In Progress
Changed in charm-aws-k8s-storage:
status: New → In Progress
milestone: none → 1.26+ck2
Changed in charm-azure-cloud-provider:
milestone: none → 1.26+ck2
Changed in charm-gcp-k8s-storage:
milestone: none → 1.26+ck2
Changed in charm-kubevirt:
milestone: none → 1.27
Changed in charm-multus:
milestone: none → 1.26+ck2
Changed in opa-gatekeeper-operator:
milestone: none → 1.26+ck2
Changed in charm-sriov-cni:
milestone: none → 1.26+ck2
Changed in charm-sriov-network-device-plugin:
milestone: none → 1.26+ck2
Changed in charm-aws-k8s-storage:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-azure-cloud-provider:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-gcp-k8s-storage:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-multus:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-kubevirt:
assignee: nobody → Adam Dyess (addyess)
Changed in opa-gatekeeper-operator:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-sriov-cni:
assignee: nobody → Adam Dyess (addyess)
Adam Dyess (addyess)
Changed in charm-sriov-network-device-plugin:
assignee: nobody → Adam Dyess (addyess)
Changed in charm-vsphere-cloud-provider:
assignee: nobody → Adam Dyess (addyess)
Revision history for this message
Adam Dyess (addyess) wrote :

upon further evaluation, adding the api loadbalancer ip to the list of hostnames in `juju model-config no-proxy` is probably the best work-around.

There aren't many python http client libraries that support cidr ranges, but only support a list of hostnames

In this SPECIFIC case, the client library used by the charm is httpx

Docs: https://www.python-httpx.org/environment_variables/#no_proxy

> Valid values: a comma-separated list of hostnames/urls

Notice that CIDR networks aren't supported here. Expanding the CIDR range here doesn't seem like a good idea, so I think that work-around is the best one to leave in place -- maybe even close the bug as a "Won't fix"

Reversion Patch: https://github.com/canonical/ops-lib-manifest/pull/14

Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

Per comment 4, changing the underlying lib that causes all these projects to be affected won't solve the issue. Given the workaround (explicit fqdn/ip in the model-config no-proxy), I'm closing these as won't fix.

We'll do a better job of calling out this scenario in our proxy docs to help future travelers debug their proxied envs:

https://github.com/charmed-kubernetes/kubernetes-docs/issues/749

Changed in charm-aws-k8s-storage:
status: In Progress → Won't Fix
Changed in charm-azure-cloud-provider:
status: In Progress → Won't Fix
Changed in charm-gcp-k8s-storage:
status: In Progress → Won't Fix
Changed in charm-kubevirt:
status: In Progress → Won't Fix
Changed in charm-multus:
status: In Progress → Won't Fix
Changed in opa-gatekeeper-operator:
status: In Progress → Won't Fix
Changed in charm-sriov-cni:
status: In Progress → Won't Fix
Changed in charm-sriov-network-device-plugin:
status: In Progress → Won't Fix
Changed in charm-vsphere-cloud-provider:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.